Quantcast
Channel: Kurt's Weblog

AIS talkers for NMEA VDM/VDO messages

$
0
0
I got an email with this question:
In my data feed m getting some sentences with packet types I do not
recognize.  Could you tell me what they mean?

!BSVDM
!ABVDM

I'm also getting multi-sentences with different packet types. Is that
allowed? First part can be ABVDM and second part AIVDM.
I was just asked in email to explain the first field in the CSV that is the AIS NMEA armored messages. Sadly, the NMEA specification is paywalled by the annoying folks at NMEA so I can't share with you where talkers are defined. The same goes for the IEC standards that define AIS device messages and testing procedures. And I only have draft versions of a few of those that people on the commitees have given me over the years. For example, I have "Maritime navigation and radiocommunication equipment and systems - Digital interfaces - Part 100: Single talker and multiple listeners - Extra requirements to IEC 61162-1 for the UAIS".

Back before ITU 1371 was opened up, I helped Eric Raymond (ESR) a bit with his writing of AIVDM/AIVDO protocol decoding. I still think that his document is drastically better written than the crap that is in the ITU documents. ESR also has NMEA Revealed with a list of talker ID.

Let's start with a sample message:
!AIVDO,1,1,,B,403OvpQuUwgBMJRe6VE5Aai005q8,0*6F,b003669730,1249053583
The first two letters after the bang ("!") are the "talker." In NMEA speak, this is the device that generated the text in your local environment. It has nothing to do with any remote AIS devices and what they sent over the VHF radio. Above, we have "AI". This is the classic "AIS" device talker. It is what you get with most AIS receivers and transceivers (they are NOT transponders).

Following the talker are 3 letters that define the "sentence type." In the world of NMEA, this is the type of message that you are about to parse. VDO is a type that not many people know about. It says that the message that follows is an AIS message that is about "own ship." This really means that the device is talking about itself. The more commonly observed VDM means "AIS message that the device received over a VHF radio channel." The VDO in the above sample message makes sense as this is an AIS basestation telling us the message 4 (basestation report) for itself. The message has two extra fields on the right: ",b003669730,1249053583". This is the older USCG logging format and has a receiver identifier and UNIX UTC timestamp. The b at the beginning of the receiver name says that the message was logged from a basestation. If we decode that message, we should find that the MMSI of the basestation report should match the receiver name.

This next sample message has "BSVDM". The talker is "BS". This means that we are hearing from a basestation. The logger has added ",r003669717" with an "r" for receiver, but it might just be missconfigured. This message is from my 2006 notes and I was just figuring out AIS and what the USCG was doing back then. I did have a basestation on my desk for a while.
!BSVDM,1,1,,A,85MwpwQKf7sgdeioePgqI0@Q;F=0q3va1sdLFPCS4wB7PtDMD:62,0*34,r003669717,1165850344
And a final sample that has "ANVDM". This was an AIS ATON (aid-to-navigation) device that uses "AN" as the talker:
!ANVDM,1,1,,B,35MnbiPP@PJutdlH2M1sIq:80000,0*10,r003669947,1294012804
As for "AB", I was unsure. I did receive some sample data with an AB talker, but I do not know what kind of device was doing the logging. AB might stand for Ais Basestation in someones mind. Looking at NMEA's closed documentation (yes, I paid rediculous amount for version 4.0 of the spec), I found:
AB Independent AIS Base Station
AD Dependent AIS Base Station 
AI Mobile Class A or B AIS Station
AN AIS Aids to Navigation Station
AR AIS Receiving Station
AS Limited Base Station
AT AIS Transmitting Station
AX AIS Simplex Repeater Station
SA Physical Shore AIS Station
There was no "BS" talker in there. Don't ask me to explain the differences between all those codes.

Please, please, please: If you have the opportunity to talk to people at NMEA, IMO, IALA, IHO, ITU, or IEC, tell them that closed / paywalled specifications are the work of the devil (and contradictory to the mission of SOLAS). It's bad enough that people aren't willing to put the discussion leading up to these specs online with their names attached. Can we please at least get ALL specs for these public data standards (because they are mandated by law in many cases) out there for free distribution licenses?!?!?

Rebecca Moore at the White House - Google Earth Engine

Disabling autologin on a mac from the command line

$
0
0
Just did this remotely on a mac. I had historically left autologin running. This was blocking some system admin tasks. Here is what I did to fix it.

http://www.cnet.com/news/how-to-disable-automatic-log-in-via-the-command-line-in-os-x/ and http://superuser.com/questions/40061/what-is-the-mac-os-x-terminal-command-to-log-out-the-current-user
ssh myhost
ps -aux | grep $USER
24
defaults read  /Library/Preferences/com.apple.loginwindow autoLoginUser
schwehr
sudo defaults delete /Library/Preferences/com.apple.loginwindow autoLoginUser
sudo osascript -e 'tell application "System Events" to log out'
ps -Ajc | grep loginwindow | awk '{print $2}' | sudo xargs kill
ps 

More open sensor format discussion

$
0
0
Originally posted here: https://github.com/tkurki/navgauge/issues/4

Kees Verruijt pinged me discussion and suggested I should add my $0.02 on requirements. Interesting discussion. For me, JSON is nice to have on the side, but I can't really consider a text based message as the primary format. I've been thinking about how to replace things like Generic Sensor Format (GSF) and the zillion vender binary formats for logging multibeam and sidescan sonars, LIDAR (e.g. tons of work in the PulseWave project), high rate intertial navigaton systems (IMU/INS), the full working set of what can be expressed for smaller sensors with NMEA 0183 messages, AIS, Radars, imagery, carrier phase GPS (aka RTK, Rinex format,...), acoustic recoders (hydrophones and geophones), e&m, an open version of MISLE ship incidents, whale sitings, data license, software versions used, the whole metadata world (ISO paywalled disaster), etc. And then think about all this combined for a very large number of platforms... e.g. all AIS receivers in a network, all buoys, AUV, ASV, etc in a fleet. On some systems, either because high resolution timing is key or because they are underwater for long periods of time without GPS (AUVs or dataloggers that may sit on the sea floor for months), multiple time sources may be essential. And I would love to see more at the OpenROV, OpenCTD, Arduino/Raspberry Pi level for cheaper and smaller sensors. We then as a community need the ability to conflate data over long time series. e.g. Maritime Rule X changed on July 1st, 2007... how did the env change in the years before and after? Did that increase or decrease regional fuel use, ship incidents, whale strikes, ship noise, economic activity at local ports, etc? I'm already see streams that are GB to TB per day and need to scale up from there. For the browser, JSON is great for small things, but the overhead of strings to binary for rendering of say just 10M points in Chrome, forced a switch to binary over the wire.

Even just the ARGOS float project has the potential to explode... they only have a few sensors right now, but what happens when they get tons more sensors and multiple routes for data to make it to a usable datastore.

I've been worrying about this for a long time now. I saw the issue with the number of sensors on the Healy all trying to spew NMEA 0183-ish stuff to NAIS with the USCG to working with sonars. I tried to start writing some of my ideas here: http://schwehr.org/blog/archives/2014-02.html#e2014-02-18T09_06_12.txt (and other blog posts). However, I haven't gotten very far. I tried making a message definition language of my own to support AIS, but did not get very far in the RTCM community. My current thought is that the best core definition place to start is with Protocol Buffers (ProtoBufs) and RecordIO or similar. That would allow for other serializations, but start off with a well defined message content that has a build-in binary serialization combined with the ability to add other serializations (e.g. JSON, XML, proto ascii, etc) to the mix fairly easily. I'm also realizing that there is a need for a journalling style system (append only) that could allow for index messages to be written to log files that would make access to logs much faster. This becomes especially important if you are later trying to join chunks of logs that might come anywhere from near realtime to months after the fact. There are plenty of other issues in there... e.g. nothing seems to talk about recording the calibration process in sensor log streams or things like time from which source and how accurate is it? And what license is the data released under? The bathymetry attributed grid (BAG) format contains ISO XML metadata, but they only have "classified" / "non-classified" for data. What about public domain, proprietary, creative commons license X, etc. ?

I think any spec has got to be totally open and it would be best to be machine readable and usable for all for any purpose. e.g. paywalled or GPL are a real problem. Something like the Apache 2 license seem like the only way to go if we really want to open up data interchange.

I've spent a lot of time trying to survive the existing disaster of standards and want to be moving towards a world in which data is easier and more fun to work with from small devices to global scale fleets of sensors. Hopefully some of these thoughts are useful to the discussion you all are having here.

Some links to material that might be interesting for you all:



P.S. GSF is now homed at Leidos (which SAIC spun off) here: https://www.leidos.com/maritime/gsf (and still not yet under an open source license).

Confusion over SRT and ExactEarth's press release

$
0
0
I posted this yesterday on google plus.

TL;DR Probably one of: it's NOT AIS (using some other freq/comm encoding), it's an out of spec Class B AIS, or a spamming binary message with EEC to compensate for expected BER. I strongly dislike that they do not say.

I'm going to have to call BS on this press release from exactEarth. They do not say what exactly their technology is in a ground based class B transceiver is that makes it better for being able to be received by satellite. If they are truly altering how they transmit class B and somehow embedding more information, then these AIS Class B units are out of spec and you should expect to have them confiscated if you use them in any countries waters (even if it's a mode that flips on/off). My best guess is that they are using a frequency owned by someone else and transmit on that with some other system. So... it's not ITU 1371.1-4 AIS at that point. Lots of other people do tracking in other frequency ranges and do that via satellites. A quick example... SPOT. I really dislike announcements like this that are all unsubstantiated hype. I don't expect system definitions in a press release, but they could reference docs frequency allocations, tx methods and power.

Or am I missing something huge? Are they fiddling with the encoding, power shaping each slot datagram, phase shifting the transmission to increase SN/correlation?

How do they get this extra information in? That implies that they are using an un-approved message type. Or they could be using a Application Specific Message (ASM)/Binary Broadcast Message(BBM) defined by some country? Then they might just spam the channel with lots of really small packets at high transmit rate. The obvious solution is to use Msg 6, 8, 26 or 27 and just fill the channel. If you ignore the AIS checksum info and have the payload contain really robust error correction data, you don't have to worry about trying to get a correct decode. Just save the data from all the known time slots and merge recoverable chunks until you have the full message. If I knew that a particular AIS unit was going to transmit the same message N times starting at know times and each TX had error correction built into the bit stream, I could definitely get the chances of receiving them message to near 100% for each satellite pass. You combine that with phase discrimination (those ships infront of the sat are shift up in freq and those behind are lower in freq), and you have a pretty decent receive rate. Being out in international waters, maybe you can get away with stuff like this. Try this in US waters and you should expect to have a visit from the feds if you are in the AIS frequencies (assuming that they can get their tracking act together). This is what a lot of other communications protocols have done for ages. It's sad that the creators of AIS never put any of this into the AIS core spec. But maybe we should be glad they didn't, because if it mirror the way other AIS specs were written, it would be an unimplementable disaster.

There are a few other possibilities, but those are the most likely.

http://exactearth.com/news/2014-05-23/
http://softwarerad.com/uploads/file_upload/file/260/ABSEA%20Whitepaper%20Draft%204.pdf



Additionally quotes like this make no sense to me. TDMA doesn't make the job of tracking a particular ship easier, but it in no way prevents you from receiving a non-Class A (e.g. Class B) message.
the TDMA radio access scheme which makes AIS such a robust
system, means that satellite detection of terrestrial transmissions
has been limited only to Class A transceivers
And if 95% of the AIS transceivers out there use SRT as their core, we have a monopoly. I need to take a look at the Class B vender field to see what the distribution is these days.

I argue that we should avoid proprietary solutions and without more info, it appears that the ABSEA thing is likely very much a closed proprietary system. I look forward to SRT publishing more information and and open (and patent-free) standard that all can implement (both tx & rx). We really should have a message for the high seas. The minimalistic AIS satellite message isn't great. Would be so much better to have a standard message for the high seas (low load VDL areas) that has EEC and transmits on a known slot. e.g. We could have a hash that converts an MMSI to a particular start slot. And if you know that a number of repeats will give the same exact message, you really have something to work with.

The current receiver design seems fundamentally flawed in requiring all messages to pass the checksum reporting. I wish that receivers would optionally report bad messages with a flag detailing the failure.

IOOC DMACS

$
0
0
Today is the start of a multi-day Interagency Ocean Observation Committee - Data Management and Communications Steering Team (IOOC DMAC) meeting. Sadly, I've always been a remote attendee as the rep from Google. I've been struggling with the question of how do we all, meaning the entire community, and especially the one actor I can control - my self, have the greatest positive impact on the future of the ocean sensing / observing community. It's a challenging question. All participants were asked a bunch of questions beforehand.

Sometimes people feel that they are not some big powerful organization and that they have no influence on things. As a grad student and as a professional, I have felt the weight of bureaucracy and the in-elasticity of people have been doing things one way for years. What can each individual do? Turns out that any one person can do quite a lot. Here are some thoughts and I'll try to make them actionable.

  • Try to stay positive in interactions with others. Technology and science can be frustrating and confusing. Try not to let that creep into interactions with others. If someone flames you, do not flame back. Especially when doing things like writing a bug report or a response to a bug report. On the other hand, do not let that keep you from being direct.
  • Report bugs in any software or hardware that you use (be it open or closed). Strive to write clear bug reports that make it as easy as possible to figure out what version of things you are using, what are the symptoms, and if you think there might be a solution, try to explain what it might be. Include a picture, screenshot, code, or data sample whenever possible. I've gotten lots of "It doesn't work" bug reports and then I have to ask back, "What does not work and how is it not working. Whenever possible, have these bug discussions out in the public where others can learn from it. If search engines can't find the info, then there is little value in it.
  • Learn to program and read open source code. Maybe you never will feel comfortable submitting a bug fix back to a project, but hopefully you will. Even if it is a spelling fix in a comment, your help is immensely important.
  • When writing code (even small couple line programs), use a style guide and write clean code. How you create data and write code is just like formal technical writing. You might be annoyed by some of the things in those guides, but they are in there because other people have long suffered with bad code and have specific reasons for almost all of those entries in the guides. e.g. While it may be cool to have single letter/symbol variables and operators in the land of mathematics, but that does not work well in software.
  • Really dig into taking notes as you work, put more energy into writing up the methods for you use, and get them out to the world. So few papers out there (including my own) have enough info that someone else could get the same results. It's hard, but we have to strive for it.
  • Stick to open data formats and standards whenever possible. Design by committee sucks, but design by yourself in a closet is just as bad. If you have to make something new, get others to review it who are domain experts before committing to strategy. You don't have to take there advise, but you should ponder what they are saying.
  • Be willing to admit mistakes and errors. If we all were perfect, then we wouldn't be humans. Improvement and learning go best when we use mistakes and errors for learning. Think constructive criticism and avoid the destructive type.
  • Learn from everyone. Everyone is accustomed to newer people learning from experienced people, but it really is a two way street. People who are new may find a different and possibly better way. Or the trouble they have may inform the experienced folk as to what needs to be changed or better explained.
That's enough for now. I'm on the call as I finish writing this and thought it was great to hear someone talk about file format advice. For that person, I have this as another data point: toils-of-ais

Global Ship tracking on the Google Cloud at Ocean Agenda

US fishing and passenger exemption for greater than 65 feet

$
0
0
I didn't realize that this exemption is in the USCG AIS Carriage Requirements (emphasis added):
- 164.46 Automatic Identification System (AIS).

    (a) The following vessels must have a properly installed,
    operational, type approved AIS as of the date specified:

     (1) Self-propelled vessels of 65 feet or more in length, other
     than passenger and fishing vessels, in commercial service and on
     an international voyage, not later than December 31, 2004.
So, if you were a 1000 ft fishing ship in US waters, you don't need AIS. Not that there are any fishing ships that large. Any why passenger vessels? It also looks like passenger vessels also get a pass, but if they enter a VTS area and have 151 or more passengers then they have to have AIS. So weird. These rules are so poorly written. Why in the world do we still have the dates in the rules when those starting dates are 10 years ago? I guess the USCG has no concept of version control.

AIS REQUIREMENTS - Title 33, Code of Federal Regulations - 164.46 Automatic Identification System (AIS).

Looking at large diffs

$
0
0
Here is a little Unix command line trick. I've been looking at the differences between version releases in software like GDAL, HDF4, HDF5, etc. There is often a lot of noise (aka non-functional changes) that get in the way of my trying to understand what is different. The diff tool is pretty useful and I often look at diffs in emacs and appreciate the highlighting of Diff mode. However, I'd rather not have this noise. In the case of GDAL, that is a lot of copyright and svn Id changes. Those don't change the bevior of the code. Here is how to get them out using the diff ----ignore-matching-lines (-I) flag. Just repeat the flag for each thing you want to not get used for diffs. e.g.
diff -ruN gdal-1.1[01].0/port -I 'Copyright' -I '$Id:' > port.diff
That says to recursively produce "unified" format diffs with the contents of new files shown, but leave out lines with "Copyright" and "$Id:'. The two -I flags drop the diff file from 7500 lines to 6200 lines for the changes between GDAL 1.10.0 and 1.11.0 in the port directory. Every little bit of help makes things easier.

Argo Floats

$
0
0
Earlier this year, Michelle made a super cool vidoe about Argos Floats.



The floats are popular and there are other videos out there.