f.zz.de

f.zz.de

Deployment rant

Posted Sun 03 Mar 2024 02:11:40 PM CET Florian Lohoff
in

For a very long time i have been running a tasking manager 2 instance for me and the regional community to fix and revalidate stuff around here. TM2 had a very short and reliable documentation and you could set it up in 5 Minutes without ANY deeper knowledge of python, node or postgres.

I had been staying at TM2 as all newer versions failed for me to build or deploy easily.

Today - another try - current git TM.

And it fails again.

It does not build on Debian/Bookworm as it is not compatibly with python 3.11 which is the obvious default version on Debian/Bookworm.

So fixing that in pyproject.toml and setting the

requires-python = ">=3.11,<3.12"

And then running

pdm lock --update-reuse

The documentation even misses the point in better creating a venv which i obviously did beforehand. Then i have the backend installed and when you get the idea that your postgres database needs the postgis extension enabled and also change your TM_LOG_DIR= to something writeable (I dont have a /home/appuser) you can populate your database with flask db upgrade.

Then going back to trying to build your frontend you need to make sure to install yarnpkg on your system. Debian has a yarn binary from cmdtools which is confusing at first. Then replace all yarn executions by yarnpkg.

Then you get to the point where the frontend starts to build and then horribly fails because of npm requirements beeing outdated and failing to build with newer openssl having fixed issues:

Error: error:0308010C:digital envelope routines::unsupported

When you try to look for it people suggest to downgrade nodejs which is:

"We have fixed a security issue and your project does not build? Downgrade to a version pre-security-fix"

WTF? This hipster tool bubble is so doomed.

As i am a sysadmin, C, C++ and perl programmer and not into JS, npm, yarn, nodejs, react and whatever "modern tools" are beeing used, i stopped here once again.

And NO - docker is not the solution when the requirement is to downgrade to less secure versions of the dependencies and just hide it away. How are you going to fix these issues in your production environment other than ignoring it?

Baujahr des Hotels

Posted Sun 25 Feb 2024 06:04:57 PM CET Florian Lohoff
in

Das Baujahr des Hotels kann man am Fahrstuhl erkennen. Besser ist aber das Telefon.

Löcher im ALKIS Schleswig-Holstein

Posted Thu 22 Feb 2024 04:35:57 PM CET Florian Lohoff
in

Nachdem ich aus dem OpenData Portal für Schleswig-Holstein ~17000 Dateien gezogen habe und das ALKIS lokal in einer PostGIS wieder zusammengepuzzelt habe fällt auf das es spannende Löcher im ALKIS gibt.

So sind die militärischen Flughäfen Jagel und Husum-Schleswig nicht im ALKIS Export mit vorhanden. Aber es ist mitnichten so das alle militärischen Anlagen ausgenommen sind. So ist die "Wehrtechnische Dienststelle 71" in Kiel komplett vorhanden.

Jedenfalls habe ich jetzt einen Hausumring export und kann das Alkisdiff SHS produzieren:

https://osm.zz.de/dbview/?db=alkisdiff-shs&layer=buildingnotinosm#54.4581,9.52897,15z

Lübeck

Posted Wed 31 Jan 2024 07:52:37 PM CET Florian Lohoff
in

Lübeck ist sehr hübsch. Sie haben auch noch eine große Menge Gaslaternen die auch sehr schön sind.

Micromapping will kill OpenstreetMap

Posted Tue 23 Jan 2024 10:06:15 PM CET Florian Lohoff
in

I have been with OpenstreetMap for 15 years. I started with an empty white nothing and early decided on mapping in a very systematic way like "All maxspeeds", "all building", "all traffic lights" and did that early with tools like Mapillary by first making Photos of all streets.

I also started using the data for civil engineer planning for Telecom infrastructure. For traffic simulations and other interesting purposes.

For 10 years i am running Quality Assurance for routing by calculating ~250k routes every 30 Minutes and watch for temporal changes in those routes.

So i guess i have a deep understanding of tagging schema. I think in hierarchies and dependencies of objects and their tagging.

Lately (for the past couple of years) i see people in for example Germany going into Micromapping. Every little stone, hedge, light, knob or straw needs to be mapped in extreme detail. Per se i dont see a problem in that undertaking.

The problem that now arises all over the place is that established tagging schemata are getting "abused". For whatever reason half-way usable tags are beeing re-used for micromapping for "physically similar" objects although these objects have a completely different sematic meaning.

This post was motivated by the abuse of highway=traffic_lights. Pure pedestrian crossings have been described with a highway=crossing crossing=traffic_lights for ages. Now people start to map additional highway=traffic_lights at every simple pedestrian crossing. So we went from a simple node to a node triplet with additional highway=traffic_lights.

As these 2 crossing types have completely seperate semantic properties this makes whole crossings indistinguishable from pure pedestrian crossings.

As routing profiles make use of these different semantic properties in estimating the delay of a traffic light this basically breaks these estimations, and worsens routing.

Turning on to a discussion on why people start doing that always ends in

the data consumer should fix his software

without really explaining how people are going to put a tangent to a point. Once removed information can not be brought back.

As a resume:

Micromapping by abusing existing tags breaks semantic meaning of data.

I think in the long run this "mapping for the renderer" and missing thinking in semantic meaning will "kill" Openstreetmap, or better, reduces the data usability for something else than a "pretty, colourful map".

I also think think this attitude of "somebody else must fix it" is also the best argument for the existence of Overture Maps.

Piaggio APE Abenteuer

Posted Tue 23 Jan 2024 05:28:11 PM CET Florian Lohoff
in

Nachdem die Temperaturen nicht mit "draussen schrauben" vereinbar waren ist Henri in die Garage umgezogen. 3 Tage quasi non-stop Dinge instandsetzen. Bremsen, Ölwechsel, Nebelschlussleuchte, Kennzeichenbeleuchtung, Armaturenbrett etc.

Dann schnell zum TÜV. Eigentlich sah es auch gar nicht so schlecht aus bis auf das die Fahrzeugidentifikationsnummer fehlt. Es gibt ein Typenschild, es gibt Papiere, aber da wo die Nummer eingeschlagen sein sollte ist ein Reperaturblech.

Ein paar hilfreiche Schrauber von Weka konnten dann weiterhelfen und eine neue Nummer einschlagen, nur fehlt die offizielle Bestätigung das das wegen Reperatur woanders eingeschlagen werden musste.

Also gibt es noch keinen TÜV.

Jetzt geht's erstmal an das Garage aufräumen

RouteQA: Check your barriers

Posted Thu 28 Dec 2023 04:33:47 PM CET Florian Lohoff
in

When putting in barriers like this barrier=bollard you might want to go back and check the routing.

In this case at least we as OpenStreetmap using OSRM send through traffic through this farmyard.

Why lanes and lane_markings matter a lot

Posted Thu 28 Dec 2023 11:33:39 AM CET Florian Lohoff
in

I am currently extending my RouteQA slowly around Northrhine-Westfalia in Germany. I now extend to areas where mappers extensively mapped lanes=1 instead of lane_markings=no which breaks routing in a lot of subtle little ways at least for OSRM.

lanes=1 means it is not possible for two way traffic to pass each other without going to the curbs or shoulder. This causes the default OSRM car profile to assume half the average speed of the maxspeed.

So a road with maxspeed=50 and lanes=1 without any oneway tagging is assumed to have a max average speed of no more than 25km/h. Thats a pretty good assumption if its really too narrow.

Now if you have a tertiary or better road in city limits with a maxspeed of 50, and a parallel road with 30km/h the smaller side roads suddenly is faster because 30km/h is better than 25km/h.

So please stop tagging lanes=1 for roads which are wide enough for passing traffic without getting into each others way. The correct tag is lane_markings=no

In the default OSRM profile lane_markings=no has no penalty at all which IMO also needs fixing which why i opened an issue for the OSRM backend

Nevertheless - be careful with lanes=1 - its a hefty penalty for routing.

This is an example of broken routing in Borken near the Dutch border from my QGis view on the RouteQA.

As one can see the Butenwall is used but also the parallel Wallstraße. The Wallstraße would be used in ALL cases if it would not be a oneway.

The Butenwall as a secondary should be MUCH better - but it (was) tagged with lanes=1 as there are no lane markings. As a secondary it is wide even for trucks to pass each other without issues.

Weihnachten 2023

Posted Mon 25 Dec 2023 12:47:25 AM CET Florian Lohoff
in

Erst ganz ruhig und minimalistisch mit Rhea und Dad ... Ein Rotwein zur Feier des Tages.

Dann Turmblasen und Weberei, wobei letzteres eher mit Jungvolk voll war.

Routing: When extensive tagging is still not enough

Posted Wed 13 Dec 2023 08:46:24 PM CET Florian Lohoff
in

In my RouteQA initiative i mostly care about temporal changed routes, e.g. routes which change because mappers make changes to the map data. I try to catch mistakes early e.g. reversed oneways, broken turn restrictions, access restriction and the like.

When i add "clusters" e.g. areas i check routes within, i also try to fix the most obvious routing mistakes e.g. shortcuts or rat-race roads. Almost all cases are caused by sparse tagging. Mappers focus on the higher class roads which carry extensive tagging about maxspeed, lanes, lane_markings, surface and the like. All the small side or rough roads lack these information and suddenly the side roads are mathematically better. This is probably true for a single vehicle, but considering we direct most of traffic to these smaller roads these roads will never cope with the traffic we send.

The first thing is that i fix the sparse tagging by iterating on the side roads and fill the gaps. In 98% of the cases this will fix shortcuts or rat-race issues. But there are hard to fix cases i up to now had no clue how to fix.

I started a thread on osrm-talk and the routing mailinglists which basically returned "unfixable", "you need to fix the engine", "the side road is better so what", "this is easy" or "unfixable without mobility or live data".

I don't buy the "mobility or live" data as thats not for hobbyists, or non profits. You need money to buy and its a huge undertaking to process that huge stream of data.

The other options are assuming "something magic happens".

So i sat down and made a "Proof of concept" inventing a tag for customizing the route weight and using that in an OSRM profile.

The tag i used is route_weight_factor:motor_vehicle which shall contain a float from -0.8 to +0.8.

This then will be multiplied with the calculated weights from the physical tags like lanes, lane_markings, maxspeed, road class etc.

So a -0.8 means we go down to 20% of the speed of the road.

Here is the git commit for the car profile change:

https://github.com/Project-OSRM/osrm-backend/commit/c199019fefc8b0d31e45a6599c0a7f1343af2cfe

This enables to fix issues like this:

The small side-road is barely wide enough to fit a car. Sending 2 way traffic through there is definitely not an option. There is no legal restriction so we cant really exclude it. So we need want to make it worse than the way around.

It shows on how mappers would be able to influence and fix routing in case something like this would get an established tagging.

Also it would allow us to respond to public bodys e.g. city councils asking for routing changes in OpenStreetMap.

Geometry antipattern: Background filling

Posted Mon 04 Dec 2023 02:24:20 PM CET Florian Lohoff
in

Continuing the OpenstreetMap antipattern blogpost series here is another one.

This is a single landuse, spanning 12km. It follows streets in a tree like shape.

As for the obvious, this kind of landuses are prone to break. People tag highway tags on them, move them around by accident, and as they ALWAYS span more than your edit area nobody really is brave enough to do something with them. So this way/landuse stayed in the Database for 12 years, despite beeing semantically nonsense.

In this case we have a landuse=grass which is in itself a broken tagging as its not a use in itself. The Wiki article about landuse=grass tells you that most likely your tagging is wrong and you should use something else.

In this case the mappers intention was to not let any gap show up on the map between roads and landuses. We know about mappers glueing landuses to roads which is a semantic error in the OSM dataset as linestring object do not have a dimension as areas do have. This is another hack to eliminate the gaps between roads and adjacent landuses.

Nevertheless this is broken. The road itself is never a landuse=grass, its not even a landcover=grass, its tarmac or asphalt. Adjacent to the road there are the shoulders, which may carry grass from time to time, or a ditch which has grass on its banks.

Concerning the tracks, these carry a surface tag themselves so this landuse just trys to map for the renderer to get gaps in the background of the map closed.

So the antipattern is to not hack OSM data to do background filling. Not with glueing landuses to linestring objects, not by filling these with huge landuses as a last resort.

As long as we dont have a language/tags to fully describe road areas (which the shoulder and the ditch belong to) we should not hack around and fake objects to close gaps.

Teileträger

Posted Fri 01 Dec 2023 11:17:39 PM CET Florian Lohoff
in

Und noch eine alte Ape als Teileträger.

Piaggio Ape

Posted Wed 22 Nov 2023 07:51:43 PM CET Florian Lohoff
in

Henri's erste Ape abgeholt. Das wird noch ein ziemliches Stück Arbeit bis das 30 Jahre alte Schätzchen wieder läuft.

Bodensee

Posted Thu 09 Nov 2023 11:44:44 PM CET Florian Lohoff
in

Ich glaube nicht das die Bodenseeregion meine bevorzugte Urlaubsregion wird. Aber im Herbst kann das schon nett sein.

Thiemo Seufer

Posted Sat 04 Nov 2023 12:58:35 PM CET Florian Lohoff
in

15 Jahre ist der Unfall her der ihn das Leben gekostet hat.

Quitte Teil 2

Posted Tue 24 Oct 2023 04:39:52 PM CEST Florian Lohoff
in

Würzburger Quitten von Henri mitgebracht wird verarbeitet.

Segfault perl with Clone::clone

Posted Thu 12 Oct 2023 06:48:32 PM CEST Florian Lohoff
in

Interesting - Using clone on a datastructure containing a ref to a dbh e.g. result from DBI->connect and using that ref from the cloned object lets perl segfault:

Short form:

$dbh=DBI->connect(...);
$a={ dbh => $dbh };
$b=Clone::clone($a);
$b->{dbh}->prepare(...)

gdb backtrace shows:

(gdb) bt
#0  0x00007f409c5e387b in XS_DBI_dispatch () from /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/DBI/DBI.so
#1  0x0000561f3f0a5f18 in Perl_pp_entersub ()
#2  0x0000561f3f09bef6 in Perl_runops_standard ()
#3  0x0000561f3effa779 in perl_run ()
#4  0x0000561f3efcc4b2 in main ()

10 years OSM routing quality assurance

Posted Wed 11 Oct 2023 12:28:44 PM CEST Florian Lohoff
in

As a surprise i today found out that its 10 years since i started doing routing quality assurance:

routeqa=> select min(osmdbtimestamp) from route;
     min
---------------------
 2013-05-28 08:00:00

So i started automatically calculating routes every 1-2 hours in May 2013 ... I now have 483525 individual routes in the database, so geometries which are "unique" and have shown up over the last 10 years.

I started with a small cluster, just 20 nodes in my hometown which i monitored for changes.

Over the years most of "Ostwestfalen-Lippe" a part of Northrhine-Westfalia in Germany was added.

Currently i monitor routes between >1500 locations in 61 clusters. So every 1-2 hours i check for changes in those routes.

Total routes calculated 46064 in 820 seconds 56.18/s

As the locations are selected by hand using QGis, its a tedious task to add them to the database. And sometimes you have a location pair which has multiple routes with equal travel time so they start flapping on every calculation. So it takes maintenance every week to keep going. Every other day you search for changes where somebody broke something by accident, mostly turn restrictions, sometimes road deletes, wrong taggings or the like.

This is just some random image/link i get when something changes. Here you see a fix of mine, taking care that through traffic stays on the higher class roads.

So .. Happy birthday ...

Wochenstart

Posted Thu 28 Sep 2023 02:47:18 PM CEST Florian Lohoff
in

Wenn die Woche erst so richtig am Donnerstag anfängt und man so kleinere Kaltstartschwächen hat.

Frankfurt

Posted Tue 26 Sep 2023 01:03:47 AM CEST Florian Lohoff
in

Das Hotel hat seine besten Zeiten definitiv seit den 70ern hinter sich. Aber die Aussicht ist gut.

Sauna

Posted Sun 17 Sep 2023 12:41:35 AM CEST Florian Lohoff
in

Erst 50°C über nacht, dann "kleine" Reparatur und eine geplatzte Kaltwasserleitung.

Eigentlich hätte nur noch Tannenöl für den Aufguss gefehlt.

Wilde Woche ...

Geheime reparatur

Posted Sun 17 Sep 2023 12:39:47 AM CEST Florian Lohoff
in

Ich glaube mir ist da was aus dem Auto gebaut worden was so 2-3l Benzin braucht. So wenig hat der noch NIE gebraucht.

Seltsames Tier in der Wohnung

Posted Sat 09 Sep 2023 11:23:33 PM CEST Florian Lohoff
in

Die KI im Staubsaugerroboter behauptet ich habe ein neues Haustier namens "Pümpel" ... Dabei weiss ich gar nicht was die fressen oder wo die wohnen.

Hinterher hab ich da so einen Menschenfressenden Nachtaktiven Pümpel.

Lüftertausch

Posted Sat 09 Sep 2023 11:21:41 PM CEST Florian Lohoff
in

Nachdem der Laptop beim booten seit einer Zeit einen "FAN error" gemeldet hat schnell bei Lenovo das Ersatzteil bestellt.

Gut ... Lenovo weiss nicht das in Europa die Telefonnummern nicht 123-123-1234 sind aber gut.

Das Ersatzteil war innerhalb von 24 Stunden da ... Respekt.

Der Lüfter ist schon spannend mit den vielen Lamellen. Getauscht ... Läuft ... Gut.

Kreativen Beschilderung

Posted Sat 09 Sep 2023 11:18:37 PM CEST Florian Lohoff
in

Bei mir direkt vor der Tür. Verkehrsberuhigt heißt Schrittgeschwindigkeit nicht 10kmh ... Aber gut.

Extinction Rebellion

Posted Sat 02 Sep 2023 07:28:01 PM CEST Florian Lohoff
in

Jetzt war ich auch mal mittendrin statt nur dabei. Alle nett. Wir hatten Zeit. Und die die gehupt haben haben alle hinten gestanden. Pech.

8er Ring-Maul Schlüssel

Posted Wed 30 Aug 2023 06:07:12 PM CEST Florian Lohoff
in

Elektronik geht wieder dafür ist die verstellmechanik auf einer Landstraße geblieben. Also die Auslasssteuerung mit einem 8er Schlüssel festgestellt und gesichert.

Ersatzteile im Zulauf.

Vitroconnect Coffee Machine

Posted Wed 30 Aug 2023 06:04:18 PM CEST Florian Lohoff
in

2011 von mir als erste Maschine (Nach einem privaten Kaffeefilter) bei der Vitroconnect gekauft. Nach 12000 Tassen verreckt mit Getriebeschaden. 2014 repariert, 2018 erneut unklar kaputt gegangen. Auf dem Hackmeeting letzte Woche zerlegt und ein kaputtes Magnetventil gefunden. Ausgebaut, entkalkt, schon geht die Maschine wieder.

Aktuell bei 19000 Tassen.

Hackmeeting Woche

Posted Thu 24 Aug 2023 04:19:25 PM CEST Florian Lohoff
in

Kaum vom Camp 23 zurück hackmeeting bei mir. Das Wohnzimmer sitzt voll mit Bastelwütigen.

Natürlich muss auch der Kaffeevollautomat erstmal repariert werden.

CCCamp 23 Tag 6

Posted Sun 20 Aug 2023 02:14:06 AM CEST Florian Lohoff
in

Das Camp ist vorbei. Abschlussveranstaltung. Die ersten Leute haben ihre Zelte abgebrochen. Wir bleiben noch bis morgen und reisen in aller Ruhe ab.

Es war heiss und schwül also war ein Bad in der Havel die beste Stelle um abzukühlen.

Nach einer Missglückten Fahrt mit der Bahn (Fahrgastrechteformular kommt) dann ein entspannter Tag. Abends noch geholfen im Milliways und bei den Saarländer die Bar leer zu trinken.