nodedb workshop in Graz

(By aaron in Confine@Funkfeuer.)

Ralf, Christian and me travelled to Graz and made a workshop on the new nodeDB. Feedback was very helpful. We have lots of new issues in github, new ideas for the UI, etc. We also explained the nodeDB API and the conversion mechanisms between old nodedb and new one.

two new chapters

(By Confine@Funkfeuer in Confine@Funkfeuer.)

Hi All,

This chapter had been a stump. It has been substantially reworked and enlarged and has received the new title
Free Networks Between Countryside and City, between North and South
The second part is completely new and contains an interview with Confine open-call-participant Carlos Rey-Moreno, Western Cape University

This chapter had perviously been part of another chapter. It has been substantially reworked and expanded. The chapter “The Incomplete Paradigm Shift” is now the political and historical contextualisation of the wireless community network movement.

With those two new chapters, the work has been more or less concluded. I aim at writing a very short kind of coda or afterword. The conclusions as such already exists, but I want to write one or two pages as a kind of summary of the summary. Maybe it will not be possible to write that in the remaining time in May but it will be done. I will also now start from the start to make corrections, add references/footnotes and links and will consider feedback received or yet to be received for smaller changes within chapters, but this is it as far as the larger structure is concerned.

I would be happy if more participants in the Confine project read this draft book. After all, the idea was to put it online in order to receive feedback, to co-write it, so to speak, with the community.

If I have learned anything from this project, then  it is all about communication.
Greetings to all and keep unwiring the world


Towards the Network Commons (Conclusions)

(By Confine@Funkfeuer in Confine@Funkfeuer.)

Hi All, this is for the time being the last chapter of my draft book on the network commons (I will now return to previous chapters and revise them) This draft chapter summarizes my findings. Based on a recent trip to Germany, where vibrant new communities have triggered discussions about what makes the essence of Freifunk, I am suggesting that the future of wireless community networks lies in the notion of the Network Commons.

Book Project Rise of the Network Commons (Update)

(By Confine@Funkfeuer in Confine@Funkfeuer.)

In April I have worked on the concept and structure of the book. I have changed the chapter structure and introduced a new sub-chapter on rural wireless and wireless community networks in underdeveloped regions.

I have conducted research on wireless community networks in poor countries and I have evaluated a number of interviews, among them Carlos Rey-Moreno, University of Western Cape, and Claudio Pisa, Rome. I have decided to use those two mostly, droping some other material.

I have done conceptual work on the whole book. This necessitated reading it from A to Z and starting to review which changes will be necessary. I have done minor edits in several sub-chapters. A final thesis is now starting to form. The thesis in a nutshell is:

Wireless (and wired) community network are forming a dispositif that is alive and well. This dispositif is not a normative descriptive model of the kind of one-size-fits all, but a flexible concept which allows different forms of realisation. A key aspect is, in which way the notion of the network commons is getting realized. If the network commons is only seen as something technical, this will create problems. In accordance with the findings of Elinor Ostrom, a commons can be successful if the community around it defines its own rules and also has ways of enforcing those rules. Guifi and Freifunk appear to be doing well because they have understood this. (This is of course much too abbreviative but will form the nucleus of a final chapter to be written after Berlin).

I have prepared my Berlin trip. I will participate in the Open Tech Summit and the Wireless Community Weekend between May 14 and 16. I will give a talk at the Open Tech Summit which will be the first public lecture where I present results from my research in the Confine context.

Optimize database performance

(By tanzer in Confine@Funkfeuer.)

In April 2015, Ralf motivated me to look at the performance of database queries. Playing with the postgresql parameter log_min_duration_statement and looking at the resulting log entries identified some queries ripe for optimizations.

  • Optimize pid_query for SAW sqlalchemy wrapper
    • Use type information, if available, to optimize pid_query
    • For unknown types, query for type_name first, then use optimized pid_query
  • Provide use_index to allow or forbid the definition of an attribute specific index
    • Add support for descending indices
    • By default, define indices for all link roles
    • By default, use indices for MOM.Id_Entity.last_cid and the MOM.MD_Change attributes parent_cid, pid, and (type_name, -cid).
  • For unary links, define the link role as unique if max_links ≡ 1

The result of these optimizations:

  • Rendering the dashboard — for the Funkfeuer Graz database — now uses a quarter of the time that was previously consumed.

  • The test scripts for stealing nodes, again for the Funkfeuer Graz database, now uses half the time.

  • Creating a database migration, postgresql to migration, now uses less than a third the time.

  • The time needed for creating a new postgresql database didn’t change at all. This affects the runtime of the converters and converting a migration into a postgresql database.

  • For IP networks, the attribute net_address needs an index.

    As the kind of mapping of net_address to database columns depends on the database used (postgresql uses a single column, sqlite uses several comlumns), this needs to be defined in the backend-specific mapper for net_address.

  • For IP networks, the attribute parent probably needs an index.

  • The dashboard queries are slowed down because they filter over both owner and manager. If we changed the object model so that only one attribute is used for filtering, the time needed for those queries would probably be reduced to less than half.

Improving localization and translations

(By tanzer in Confine@Funkfeuer.)

In March and April 2015, I improved localization and translations:

  • Ensure that the language selected by the client is used for every request
    • There were some cases where this didn’t happen before
  • Improve auto-generated ui_name for links
    • Use ui_name of the roles, if any
    • Replace underscore by space between role-name and connector
  • Add explicit definition of ui_name to some attributes, for instance, for the attribute standard of Wireless_Channel and Wireless_Interface.
  • Use localized names for dashboard sections
    • Add explicit definition of name for the dashboard section for IP addresses
  • Update translations
    • Though there is still lots of work to do there!

UI improvements of dashboard forms

(By tanzer in Confine@Funkfeuer.)

In March and April 2015, I improved the UI of dashboard forms:

  • Different color scheme for input fields with focus

  • Mark required fields more clearly

  • Avoid layout jumps when a form button is clicked

  • Remove readonly fields from tabbing order

  • Adapt and improve CSS used for jQuery-UI dialogs

    • Color scheme
    • Ensure visibility of large completion dialogs in entity-selection widgets
  • Improve attribute completion in forms

    • For attributes with a completion treshold of 0, display the completion dialog automatically when the input field gets focus
    • For multi-attribute completions, return possible choices for the single trigger attribute if the multi-attribute completion doesn’t find any matches.
  • Improve behavior and CSS of ESF dialogs

    • Mark input that doesn’t match any existing entity with class "bad"
    • Add feedback if there are too many completions to display
    • Handle multiple calls of activate_cb gracefully
      • close button didn’t work if a entity-selection dialog was activated twice (by clicking into the corresponding input while the dialog was already open).
    • Hide jQuery-UI close button in dialog title
      • The button is very small and the cursor displayed has its action point well inside the cursor shape — it’s quite hard to click the button properly
      • Move form buttons to top of dialog so that they are always visible, even when a completion popup hides the most of the dialog
    • Focus apply_button in completed_cb
    • Add class pure-button-primary to Apply button
    • Increase contrast of .ui-dialog-title
    • Remove header from entity_selector_form_p
  • Add jinja macro id_entity_select

    • Use a html select element to choose an instance of an E-Type if there are less than 100 possible choices.

    • To enable this for a specific attribute, set:

       input_widget = "mf3_input, id_entity_select" 

      for the specific attribute or its attribute-type.

  • Add and use attribute polisher compress_spaces

    The Funkfeuer Graz database contained duplicate addresses that only differed by the number of spaces between street name and street number.

  • Improve vertical alignment of checkbox in forms

  • Use type-specific edit-forms for dashboard

    For partial types, e.g., Net_Interface, use a type-specific edit-form for instances — the generic edit-form that used to be displayed doesn’t show any of the attributes added by the specific type.

    For instance, for Net_Interface, open either a Wired_Interface or a Wireless_Interface form, depending on the type of the concrete instance for which the edit button was clicked.

  • Improve handling of commit errors

    Errors triggered by predicates checked during commit weren’t displayed properly in the form; I fixed that.

  • Improve display of errors in forms

    • Add error description to aside of input fields
    • Put focus on first input field with error after a failed submit

Use user-specific cache for Admin_Restricted

(By tanzer in Confine@Funkfeuer.)

The dashboard showed the wrong nodes if several different users accessed the node-db at the same time. The confusion was caused by a cache that didn’t take into account the user triggering the request.

Once identified, the problem was easily fixed. Unfortunately, it’s still waiting to be tested and to be closed.

Adapting to Debian Jessie

(By tanzer in Confine@Funkfeuer.)

In March 2015, I adapted the node-db code to Debian Jessie.

  • In comparison to Debian Wheezy, Jessie supports a lot more of the required third party packages: adapt the README file.

  • Adapt to sqlalchemy 0.9.8

    • From one version to the next, sqlalchemy changes the SQL generated.

      From version 0.8 to 0.9, sqlalchemy changed only a little (and just cosmetics), for instance:

      • parentheses around sub-expressions of ORDER-BY or WHERE clauses (sqlalchemy 0.9 add some parentheses, while removing some others).
      • true vs. 1 in expressions

      There were two doctests in Tapyr and one the common-node-db that show SQL generated by sqlalchemy that needed to be changed accordingly.

    • Fortunately, this time there were no changes in the sqlalchemy API — in the past, classes and functions would move from one sqlalchemy module to another or change names.

  • Add support for the Debian packages cssmin and jsmin

    • In Jessie, cssmin and jsmin are available as debian packages cssmin and python-jsmin, respectively.

    • Up to now, rcssmin and rjsmin were used via pip install.

    • Now, either of these will be used if installed. Because the modules packaged by Debian are a lot slower than rcssmin and rjsmin, the latter will be preferred if both are installed.

      On my machine, the speed difference of the unit test using these modules:

      • 5 seconds with rcssmin, rjsmin (these are installed from pypi)
      • 37 seconds with cssmin, jsmin (these are packaged by debian)
    • These modules are optional but highly recommended, they are only used during deployment for minimization of CSS and Javascript files.

  • Add support for Apache 2.4

    • starts with:

      There have been significant changes in authorization configuration, and other minor configuration changes, that could require changes to your 2.2 configuration files before using them for 2.4.

      What an understatement!

    • In Apache 2.2, it was possible to have a fcgi script residing outside a virtual host’s document root. In Apache 2.4, that doesn’t seem to work anymore. Unfortunately, the upgrading document doesn’t mention this topic at all.

      “a maze of twisty little passages, all alike”

  • Not directly related to Jessie, but a requirement of the Funkfeuer Graz folks, I added support for using wsgi instead of fcgi.

Improve session handling

(By tanzer in Confine@Funkfeuer.)

In December 2014 (Late update of Confine Blog!), I improved the sessions handling.

  • Don’t save empty sessions
    • in particular, don’t open unnecessary sessions triggered by crawlers
  • Keep when a session expires or has invalid hash
    • mark session as invalid in this case
    • ignore of invalid sessions
    • increase max_age of session cookie to 1<<31
  • Don’t change on login or change of password
  • Use uuid4 for instead of randrange (big number)
  • Add request.path to Anti_CSRF.secrets