Parsing and analyzing Minecraft ore distributions

Some months ago, my 6 year old son discovered Minecraft. His sister and I soon followed1. It’s a great game and there’s a ton of good content out there about it. Reddit and forums are active. it has an excellent wiki, and lots of YouTube stuff. Not surprising given how popular the game is.

One topic that is covered a bunch is how to best mine for diamonds. The thing is that beyond a graph of diamonds by layer, I don’t really see any data involved.

In this post, I talk about how I parsed a Minecraft Bedrock world database to get some more numbers. Topics include:

  • Pictures of some fun situations and overall distributions.
  • Some numbers concerning the densities/distributions diamond, other uncommon ores and spawners.
  • How to parse the data yourself.
    • Mojang/Microsoft has released their variant of Google’s leveldb, but it doesn’t compile out of the box and it doesn’t come with a reference parser.
    • The wiki is very helpful but I find that actual code samples tell me more.

A big caveat…

This post mentions data from only one Bedrock world which hasn’t been traveled much. The seed is “-1337710146”. I don’t remember where I got it but it was from a YouTube video. Basically, I loaded it in creative and flew around a bunch to trigger terrain creation.

In the end, my data represents:

  • 3705 16*16*256 block chunks
    • so an area of 3705*16*16 or about 1000×1000
  • 87,949,312 blocks are actually stored. This is less than the 242M blocks in the world. There’s no need to store subchunks of the sky.
  • 12324 diamonds or about 3.3 per chunk.

My main point is to enable others to parse their data

My main intent in this post to to share the code I used to compute these numbers. I’d be really surprised if I don’t have a big bug in there. Don’t trust my numbers, play with the code yourself.

At the same time, I realize that most minecraft players are not computer programmers. Hopefully, the stats I give are interesting to you.

See the second half of this post for instructions on how to parse minecraft bedrock data.

My code is available on github.

For non-programmers:

Some fun pictures

In my explorations playing the game, I hadn’t come across diamonds in lava. I’d known to watch out for diamonds over lava, but not swimming in it.

In this picture, there’s a diamond behind the cobblestone block, directly below flowing lava. There’s also one to the right of it.

This is the funnest one. My program gave me a list of diamonds under lava. I went to the location and found this… where’s the diamond?

Douse it with water.

Pick out the obsidian and there it is. You’d have to be incredibly lucky to find this one.

There’s only one diamond, so that luck wouldn’t have taken you very far.

I was also a bit surprised to find some spawners in lava.

Interesting stats

Given 87,949,312 stored blocks, here’s a high level breakdown:

Block TypeNumber in my worldpercentage overallpercentage ignoring air
diamond123240.0140126%0.0182668%
lapis lazuli145890.016588%0.021624%
stone5246566659.6544%77.7652%
air2048258823.2891%30.3595%
dirt39020474.4367%5.78366%
bedrock33177023.77229%4.91754%
gravel28633753.25571%4.24413%
leaves8067370.917275%1.19576%
coal_ore7211200.819927%1.06885%
grass6956760.790997%1.03114%
iron_ore3923750.446138%0.581583%
water3888140.442089%0.576305%
stained_hardened_clay3386190.385016%0.501905%
lava3364900.382595%0.49875%
hardened_clay2259810.256945%0.334952%
sand1975840.224657%0.292861%
vine1490900.169518%0.220983%
redstone_ore975400.110905%0.144575%
log956050.108705%0.141707%
sandstone952220.108269%0.141139%
tallgrass839560.0954595%0.124441%
gold_ore529830.0602427%0.078532%
bamboo419860.0477389%0.0622322%
snow_layer271660.0308882%0.0402658%

Clustering

Diamonds, and other ores, tend to be found in clusters. If you find a diamond, you should dig around a bit as there are usually a couple more nearby. I define a cluster as a set of ores in which every member is 2 blocks or less away from another member.

In the table below, for each size of cluster, the number of such clusters is given for each ore type. In the case of diamonds, if you find one you’ll usually find between 4 and 6.

 diamond_oreiron_orelapis_oreredstone_ore
125824671931773
223525572721779
326825823411740
4566739911074030
539864305412707
6478119735142923
759402855513
8354117962632278
93320319386
10929463269
1117880200
1206961212
1315460130
1416310126
150340039
160457067
170188028
180191015
19090014
20090016
2105707
2206407
2302805
2403402
2501702
2601601
2701000
280701
290600
300401

Depth/y/altitude

We’re told to mine at y=12. Here is a table that shows where the ores are in my particular world. The heading contains the total number of each ore. If you’re looking for diamonds, you can basically look anywhere between 5 and 12.

oretype(totalinworld)diamond(12324)iron(380887)lapis(14589)mob_spawner(163)redstone(97540)
00(0%)0(0%)0(0%)0(0%)0(0%)
10(0%)0(0%)0(0%)0(0%)0(0%)
2251(2.0367%)1656(0.43477%)78(0.53465%)0(0%)2337(2.3959%)
3578(4.69%)3320(0.87165%)150(1.0282%)0(0%)4466(4.5786%)
4789(6.4021%)4885(1.2825%)276(1.8918%)0(0%)6790(6.9612%)
51140(9.2502%)6683(1.7546%)470(3.2216%)0(0%)8849(9.0722%)
61087(8.8202%)6817(1.7898%)441(3.0228%)0(0%)8882(9.106%)
71090(8.8445%)6960(1.8273%)527(3.6123%)0(0%)9007(9.2342%)
81139(9.2421%)6888(1.8084%)584(4.003%)0(0%)8770(8.9912%)
91126(9.1366%)6874(1.8047%)828(5.6755%)0(0%)8569(8.7851%)
101096(8.8932%)6683(1.7546%)862(5.9086%)0(0%)8626(8.8436%)
111114(9.0393%)6626(1.7396%)740(5.0723%)2(1.227%)8943(9.1685%)
121125(9.1285%)6717(1.7635%)810(5.5521%)3(1.8405%)9036(9.2639%)
131060(8.6011%)6455(1.6947%)914(6.265%)4(2.454%)7928(8.1279%)
14608(4.9335%)6527(1.7136%)882(6.0457%)2(1.227%)4488(4.6012%)
15121(0.98182%)6622(1.7386%)901(6.1759%)4(2.454%)849(0.87041%)
160(0%)6754(1.7732%)835(5.7235%)2(1.227%)0(0%)
170(0%)6795(1.784%)805(5.5179%)5(3.0675%)0(0%)
180(0%)6637(1.7425%)798(5.4699%)4(2.454%)0(0%)
190(0%)6664(1.7496%)682(4.6748%)5(3.0675%)0(0%)
200(0%)6611(1.7357%)550(3.77%)6(3.681%)0(0%)
210(0%)6619(1.7378%)524(3.5917%)3(1.8405%)0(0%)
220(0%)6812(1.7885%)417(2.8583%)9(5.5215%)0(0%)
230(0%)6800(1.7853%)378(2.591%)7(4.2945%)0(0%)
240(0%)6743(1.7703%)358(2.4539%)1(0.6135%)0(0%)
250(0%)6740(1.7696%)312(2.1386%)2(1.227%)0(0%)
260(0%)6735(1.7682%)203(1.3915%)3(1.8405%)0(0%)
270(0%)6700(1.7591%)132(0.90479%)5(3.0675%)0(0%)
280(0%)6633(1.7415%)93(0.63747%)5(3.0675%)0(0%)
290(0%)6623(1.7388%)37(0.25362%)6(3.681%)0(0%)
300(0%)6408(1.6824%)2(0.013709%)5(3.0675%)0(0%)
310(0%)6448(1.6929%)0(0%)2(1.227%)0(0%)
320(0%)6667(1.7504%)0(0%)5(3.0675%)0(0%)
330(0%)6649(1.7457%)0(0%)3(1.8405%)0(0%)
340(0%)6247(1.6401%)0(0%)4(2.454%)0(0%)
350(0%)6600(1.7328%)0(0%)6(3.681%)0(0%)
360(0%)6835(1.7945%)0(0%)1(0.6135%)0(0%)
370(0%)6895(1.8102%)0(0%)5(3.0675%)0(0%)
380(0%)6823(1.7913%)0(0%)5(3.0675%)0(0%)
390(0%)6800(1.7853%)0(0%)0(0%)0(0%)
400(0%)6648(1.7454%)0(0%)4(2.454%)0(0%)
410(0%)6448(1.6929%)0(0%)3(1.8405%)0(0%)
420(0%)6581(1.7278%)0(0%)2(1.227%)0(0%)
430(0%)6802(1.7858%)0(0%)2(1.227%)0(0%)
440(0%)6811(1.7882%)0(0%)5(3.0675%)0(0%)
450(0%)6905(1.8129%)0(0%)2(1.227%)0(0%)
460(0%)6769(1.7772%)0(0%)1(0.6135%)0(0%)
470(0%)6990(1.8352%)0(0%)2(1.227%)0(0%)
480(0%)5792(1.5207%)0(0%)4(2.454%)0(0%)
490(0%)5860(1.5385%)0(0%)1(0.6135%)0(0%)
500(0%)5892(1.5469%)0(0%)1(0.6135%)0(0%)
510(0%)6101(1.6018%)0(0%)2(1.227%)0(0%)
520(0%)5898(1.5485%)0(0%)0(0%)0(0%)
530(0%)5820(1.528%)0(0%)1(0.6135%)0(0%)
540(0%)6017(1.5797%)0(0%)0(0%)0(0%)
550(0%)5994(1.5737%)0(0%)0(0%)0(0%)
560(0%)6149(1.6144%)0(0%)1(0.6135%)0(0%)
570(0%)6054(1.5894%)0(0%)2(1.227%)0(0%)
580(0%)5920(1.5543%)0(0%)3(1.8405%)0(0%)
590(0%)5637(1.48%)0(0%)0(0%)0(0%)
600(0%)5158(1.3542%)0(0%)2(1.227%)0(0%)
610(0%)4162(1.0927%)0(0%)3(1.8405%)0(0%)
620(0%)2112(0.5545%)0(0%)2(1.227%)0(0%)
630(0%)416(0.10922%)0(0%)0(0%)0(0%)
640(0%)0(0%)0(0%)1(0.6135%)0(0%)
650(0%)0(0%)0(0%)1(0.6135%)0(0%)
660(0%)0(0%)0(0%)0(0%)0(0%)
670(0%)0(0%)0(0%)1(0.6135%)0(0%)
680(0%)0(0%)0(0%)0(0%)0(0%)
690(0%)0(0%)0(0%)2(1.227%)0(0%)

Caves vs mining

In various forums, there’s the question of how many diamonds will you find just by exploring caves. To find an ore in caves, it needs to be next to 2 an air block. According to my data, out of 2661 diamond clusters, 77 of them had a member next to an air block. This represents 311 diamond ores or a little over 10%.

Mining strategy

Online, there’s a lot of talk about the best branch mining strategy. Tunneling every 4 blocks guarantees seeing everything on your level and one above. But… it doesn’t take into account that if you find a desired ore, you’ll continue digging the area. Clustering.

So, I added some code to see how many clusters you’ll find with each branch mining spacing. If you find one member of a cluster, you’ll find the whole cluster.

So, the table below is kind busy.

  • The rows represent the y/altitude level of your feet.
  • The columns represent the number of blocks between each branch. So if you have zero blocks between each branch, you’re mining two complete layers. One block means, branch/block/branch/block…
  • The heading tells you the number of blocks between branches and what percentage of the two layers you’re mining.
  • The table cells have two numbers.
    • The percentage of all diamonds you’d find for that level and branch spacing. Since Diamonds are distributed over 10+ levels, you wouldn’t get them all.
    • On average, how many blocks you’d have to mine per diamond found. The numbers seemed low to me after first. 125 blocks per diamond was not my experience. Remember however that diamonds are usually next to friends. If you think of 4/cluster, that means you’d have to dig 500 blocks to find a a cluster.

Sorry that the heading font is so big. I couldn’t find a way to get TablePress to make it smaller.

In the end, it doesn’t really seem to matter much how many blocks you skip, though it does seem that digging a little deeper is helpful.

When computing these numbers, I consider that you’ll find diamonds if:

  • A cluster member is in the branch you’re mining.
  • directly above or below
  • directly left or right.
 0/ 1001/ 502/ 33.333/ 254/ 205/ 16.676/ 14.297/ 12.58/ 11.119/ 1010/ 9.091
level 217.96/ 857.214.96/ 514.410.39/ 493.68.747/ 439.96.994/ 440.15.631/ 455.65.023/ 437.84.073/ 472.43.968/ 4313.611/ 426.33.205/ 436.6
level 327.84/ 552.925.12/ 306.418.19/ 28215.97/ 24113.21/ 23310.32/ 248.69.453/ 232.68.009/ 240.26.913/ 247.46.362/ 2426.045/ 231.5
level 435.56/ 432.832.36/ 237.823.79/ 215.721.34/ 180.319.18/ 160.514.37/ 178.512.88/ 170.811.05/ 174.19.25/ 184.99.12/ 168.88.244/ 169.7
level 540.97/ 375.738.15/ 201.828.51/ 179.925.66/ 15023.81/ 129.317.8/ 144.115.55/ 141.413.59/ 141.611.69/ 146.311.77/ 130.810.29/ 136
level 645.77/ 336.341.85/ 183.932.11/ 159.827.47/ 140.123.64/ 130.219.01/ 134.918.23/ 120.614.26/ 13513.46/ 12711.73/ 131.211.09/ 126.2
level 747.03/ 327.343.29/ 177.832.43/ 158.228.81/ 133.625.02/ 12320.49/ 125.217.48/ 125.815.45/ 124.513.32/ 128.412.7/ 121.210.94/ 127.9
level 846.56/ 330.643.35/ 177.532.88/ 156.129.02/ 132.625.98/ 118.521.15/ 121.316.57/ 132.716.06/ 119.813.84/ 123.513.56/ 113.510.65/ 131.3
level 947.36/ 32543.47/ 177.133.72/ 152.128.51/ 13525.37/ 121.421.06/ 121.816.82/ 130.715.08/ 127.615.25/ 112.113.28/ 115.910.81/ 129.5
level 1047.53/ 323.943.7/ 176.132.58/ 157.529.19/ 131.824/ 128.321.14/ 121.417.15/ 128.215.27/ 12614.69/ 116.412.82/ 120.110.87/ 128.7
level 1144.81/ 343.541.74/ 184.431.45/ 163.128.22/ 136.424.69/ 124.719.82/ 129.416.7/ 131.714.82/ 129.813.92/ 122.913.06/ 117.910.16/ 137.7
level 1238.27/ 402.236.45/ 211.128.12/ 182.426.05/ 147.722.52/ 136.718.09/ 141.815.79/ 139.314.44/ 133.212.92/ 132.411.41/ 134.98.901/ 157.2
level 1329.07/ 529.427.61/ 278.721.43/ 239.420.49/ 187.816.7/ 184.314.57/ 17612.49/ 176.111.04/ 174.210.42/ 164.28.634/ 178.37.562/ 185
level 1419.99/ 770.218.56/ 414.713.88/ 369.612.94/ 297.310.97/ 280.69.226/ 278.17.806/ 281.77.23/ 266.16.142/ 278.45.721/ 269.14.998/ 280
level 1510.78/ 14278.804/ 874.26.313/ 812.85.956/ 646.14.187/ 735.33.708/ 691.83.181/ 691.33.457/ 556.62.475/ 691.12.434/ 632.32.045/ 684.3

Parsing the code yourself/the code

There are three parts to the code you may care about3:

  • Getting Mojang’s leveldb code to work
  • Parsing the actual structures
  • computing statistics

Getting Mojang’s leveldb code to work

Mojang was “kind enough” to release the leveldb code. Super helpful except it doesn’t compile for me to use it out of the box.

Why do I put quotes around “kind enough”? Because I want to go on what I think is an interesting tangent.

A tangent on open source software

I used to work at Intel Corp as a computer programmer. Intel’s design team uses A LOT of Perl code. Until 5 years ago, when I left, it was mostly Perl 5.8.5 but that was beginning to change. I believe the version was 8 years old at the time. It was mostly a matter of libraries/packages compatibility.

Anyway, I really liked Perl and I wrote a lot of it. Over time, I cooked up some packages that I thought others in the Perl community would find useful. So I contacted the folks in my division who deal with open source stuff. After one conversation with the guy it became clear to me, “Intel really doesn’t want to release anything into open source”

Intel is dependent on open source software. Design has always been done on various flavors of unix/linux. Redhat and SUSE are two that I remember.  Lots of other related software. Boost, GNU, the list is long. Still, Intel really doesn’t want to release anything.

Do you feel outraged? Don’t be. The reasons are not Intel’s fault.

Intel goes through great lengths to pay for free open source software. Intel searches for someone to pay for everthing they use. GNU… they pay someone (FSF?, Redhat/SUSE?) They pay Nokia to use QT. If you want to use a piece of OSS, you find need to find someone who will take money for “support”, but it’s not about support, it’s about paying for what Intel uses.4

But why? Intel is a large company with lots of money. If they were taken to court, it’d be easy to convince a jury that big bad Intel is taking advantage of people. So they insist on paying for everything.

During my 22 years there, I heard the following several times, “if you ever end up on the stand in court, there will be a lawyer who is much much smarter than you who will make you look like an evil idiot”

So why couldn’t I release my Perl module on github or wherever?

How do I know my module is 100% my code? How do I know I didn’t copy some code from Stackoverflow. By releasing my module, I’d (as someone working at Intel) be open sourcing someone else’s code. That someone did not give Intel permission to do that.

When I was discouraged from opening my code, it wasn’t about being overprotective about Intel’s IP. There’s very little that I know that would be of any interest to NVidia, AMD/ATI, TSMC, Apple… To get what I know, all they’d have to do is hire me. I know lots of people that have switched to those companies.

It wasn’t about protecting Intel’s IP, it was about protecting the IP of others, or more accurately, not stepping on the rights of others ourselves. Don’t want to end up on the stand against the smart lawyer.

Thankfully, since leveldb was already open sourced by Google, Mojang had no choice but to release their own changes.

As far as I can tell, the only changes they made was to add some additional compression to the format. If they hadn’t done that, I imagine we wouldn’t have gotten a leveldb from them at all. We’d have no idea which version of the code applies.

I do believe that Mojang wants to be open. It’s core to the philosophy of Minecraft.

End tangent

Getting leveldb-mcpe to compile/work

Although it burned a bunch of time figuring it out, the answer is easy. Two things:

  • The existing code includes <snappy/snappy.h>. On my ArchLinux system, I need to include <snappy.h> instead.
  • There’s a file table_test.cc which wants the function port::Snappy_Compress, but it’s missing. Just comment that line out. It’s only benchmarking stuff and doesn’t matter if you’re just trying to parse a minecraft world.

Stupid easy, but my experience as a programmer didn’t let me believe it at first. I searched for old versions and figured that if one thing’s missing, what’s next.

Parsing the actual data

Big/little endian

On the minecraft wiki, several mentions are made towards both big and little endian, but all of the data I came across is the same format. I use this function to extract all 4 byte numbers:

int32_t get_intval(leveldb::Slice slice, uint32_t offset)
 {
     int32_t retval = 0;
     for(int i=0; i<4; i++) {         // if I don't do the static cast, the top bit will be sign extended.         retval |= (static_cast(slice[offset+i])<<i*8);
     }
     return retval;
 }
NBT Data

All NBT elements have a name. That name is often a zero length string, but there’s a name. No big deal.

NBT strings also have a name. So basically two strings combined into one.

Parsing the rest

I won’t write anything about that. it’s either covered on the Minecraft wiki or easy to get out my code. The relevant file is this one. Note that I only pay attention to the block code. Mobs and players are ignored in the current version.

It’s not a particularly large file.

Clustering ores

To cluster the ores, I used an algorithm that’s probably overkill. I compute a Euclidean Minimum Spanning Tree. The algorithm for MST is one that CS majors learn but that algorithm requires a graph. The easy answer is a complete graph but if you have 1000 nodes, that’s a million edges. Instead you run on a triangulation. Specifically the Delaunay triangulation. CGAL and Boost libraries to the rescue.

I guess that’s it. It feels kind of abrupt, but I can’t think of additional relevant details. Comments and questions are welcome.

Next Steps

Minecraft behavior is often a mystery. One thought I have is creating a module that could occasionally run on a server that scrapes interesting data about your world and maybe presents it on a website. One example that comes to mind is village data. Where are your village centers? Perhaps you accidentally merged multiple villages. Is it cheating to find out?

Would this be interesting?

Please like, share, comment and subscribe. And don’t forget to click the notification bell.

Oh wait, this is a blog. I’d love to get comments though.


  1. I actually bought it for my daughter on the Google playstore probably five years ago, but she wasn’t interested. Now, she’s as hooked as the rest of us.

  2. “next to” does not include diagonal

  3. There are three kinds of people in the world. Those that can do math, and those that can’t

  4. Sadly, they really want to pay a company. We once tried to get Intel to pay a gdb developer to fix some bugs that were biting us. No luck.

My version of a needle based foam cutter

Some time ago, I came across this forum thread about cutting foam with a reciprocating needle.  There’s also a pretty good summary that distills a bunch of the details.

Here’s a video that talks about my experiences in getting such a thing to work:

 

Here’s a link that I mentioned in the video about working with the HC-05 module

The other stuff is all pretty easy to find with some simple searches.

 

Organizing photos with exif and more

I’ve recently spent a bunch of time organizing a pile of picture files in various directories. This post is about some of the utilities I’ve found more useful.

I want this structure:

2012/
├── 01
│ └── 31
│ ├── 2012-01-30_17-26-35_65.jpg
│ └── 2012-01-30_17-26-39_533.jpg
├── 02
│ └── 13
│ └── 2012-02-13_15-37-29_814.jpg

I often have this structure:
dir1
├── IMG_20171204_140837.jpg
├── IMG_20171204_140838.jpg
├── IMG_20171204_140839.jpg
├── IMG_20171213_085000.jpg
├── IMG_20171213_085021.jpg
└── IMG_20171228_110227.jpg
dir2
├── IMG_20171204_140839.jpg
├── IMG_20171213_085000.jpg
├── IMG_20171213_085021.jpg
├── IMG_20171213_085025.jpg
├── IMG_20171217_160410.jpg
└── IMG_20171217_160411.jpg

Best option…

if you files have create date stamps in the picture files, this command is the best option:

[code]

# Create Date : 2018:04:22 16:05:54
exiftool ‘-Directory&lt;CreateDate’ -d %Y/%m/%d -r

[/code]

This will set a file’s directory to reflect the CreateDate property. If there’s no such property, exiftool will do nothing.

Remove empty directories

After the “best option”, you may end up with empty directories.

[code]

find . -type d -empty -delete

[/code]

Remove duplicates

[code]

fdupes -rdN &lt;dir&gt;+

[/code]

This will keep the “first” of a set of copies and remove the rest. I’ve found it difficult to predict which will be first. I understand it’s by name but it often seems not to. Be careful for the situation where one copy is in a directory that tells you the date and another that perhaps isn’t.

Add a CreateDate tag

The date tag should look like this:
2018:04:22 16:50:28

You may have directory names that look like this:

2018-04-22
2008-11-01/IMG_1040_0018_018.jpg

Exiftool to the rescue

[code]

exiftool -overwrite_original ‘-CreateDate<${directory;s/ \d+$//;s/\-/:/g} 00:00:00’ \
-if ‘not $CreateDate’ -r

[/code]

The main thing to remember is that exiftool is a perl script. The stuff in the ${} is just perl stuff.

You’re just creating a string that looks like: 2018:04:22 16:50:28

The same is possible with filenames:

[code]

exiftool -overwrite_original \
‘-CreateDate&lt;${filename;s/^.*_20/20/; s/(\d{4})(\d\d)(\d\d)_(\d\d)(\d\d)(\d\d).*/$1:$2:$3 $4:$5:$6/}’ \
-if ‘not $CreateDate’ -r

[/code]

If all else fails

There are cases like with mp4s where I can’t set the date tag 1. In those cases, I just want to merge one directory set with another

[code]

rsync –prune-empty-dirs -a –recursive –verbose –ignore-existing \
–remove-source-files a b

[/code]

This copies files from a to b, deleting the source copies from a as it goes. It won’t overwrite existing files in b. Last it does some cleanup/pruning as it goes.


  1. A least exiftool doesn’t seem to want to it. I haven’t really tried to find out why

Comparing register states in embedded gdb with regview

regview is a gdb based utility for viewing control register state. ADC, DMA, RCC,… I use it to view STM32 registers

I am not the original author, but I needed some additional features. In particular, the ability to save and compare states from two sessions. I had a reference prototype that worked and some code of my own that didn’t. Compare register states between the two helped me move forward.

 

 

The enhancement I talk about are here:
https://github.com/mmccoo/gdb-regview

It is a fork of this original:
https://github.com/fnoble/gdb-regview

It uses register definitions from here:
https://sourceforge.net/p/embsysregview/code/HEAD/tree/trunk/org.eclipse.cdt.embsysregview.data/data/cortex-m3/STMicro/

gdb python is documented here:
https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html

 

 

Understanding and parsing SVG paths

I recently had the need to parse vector paths from an svg file generated by inkscape. I wanted to convert some graphics into kicad zones and/or boundaries. So I drew some stuff in inkscape and looked at the resulting svg file. Didn’t look so hard to parse.

Turns out there are a fair number of details to consider:

  • The path format itself. relative and absolute coordinates, closing polygons…
  • Transformation matrices embedded in the svg.
  • Global coordinate space. I want the graphics to be a specific size on my PCB.
  • Polygons can have holes. Inkscape doesn’t associate them for you. It doesn’t tell you which polys are boundaries and which are holes 1

You can find the python code I came up with here. It implements the svg parsing functions that I needed to turn this in inkscape:

Into this in kicad2

This post only talks about the svg stuff. For the kicad side of things, go to my kicad.mmccoo.com blog

The basic path

There are lots of online descriptions of svg paths. Like this one. The basics:

[code]

<path d="m 280.53386,834.58414 0,91.42158 z m 18.65215,23.8483 31.17691,0 z" />

[/code]

  • A path is stored in the ‘d’ attribute of a path node
  • A path begins either with a m or M. In both cases, it says to move to a specific, absolute value, point.
  • Subsequent points, after an m/M are line commands.
    • if it was lower case m, the points are treated as relative to the previous position
    • with upper case M, the points are absolute.
    • in the example above, we have a line 280,834 to (280+0),(834+91)
  • the letter z says to draw another line to the previous m/M.
  • The second m, is a relative move command to (280+18,834+23). Remember, because of the z, we’re back to the original spot. Also, subsequent points will be lines with relative positions.
  • The second z closes with a line to the previous m, in this case (280+18,834+23)

There are other commands.

  • L says to draw a line to the following absolute coordinates
  • There are others, H (horizonal), V (vertical), q, quad curve. These I don’t handle in my code.

Transformations

The svg I got from inkscape, has some transformations. With some stuff deleted:

[code]

<g transform="translate(0,-698.0315)">
<g transform="matrix(1.5568383,0,0,4.682378,-27.754271,177.53645)">
<path d="m 261.51562,147.55469 0,87.8125 z" transform="matrix(0.29893589,0,0,0.13478681,50.825779,127.66697)" >
</g>
</g>
[/code]

For a description of svg transforms, I found this page helpful. SVG transformations are represented in matrix format, which supports rotation, movement, and scaling. Here is a good wikipedia resource.

So in the example above, each of the path’s points first need three transformations, bottom up. Two of those transforms are given directly as a matrix. The top one, is a simple translation that is easily converted to a matrix. To make this easier, I elected to multiply the three matrices together by applying this function a couple times:

[code]
def multiply_transforms(a, b):
# this assumes a and b are represented with these indexes
# 0 2 4 <- these are indexes of a and b
# 1 3 5
# "0" "0" "1" <- these always have the value 0 0 1
retval = [
a[0]*b[0]+a[2]*b[1], # 0
a[1]*b[0]+a[3]*b[1], # 1
a[0]*b[2]+a[2]*b[3], # 2
a[1]*b[2]+a[3]*b[3], # 3
a[0]*b[4]+a[2]*b[5]+a[4], # 4
a[1]*b[4]+a[3]*b[5]+a[5] # 5
]
return retval

[/code]

With this computed matrix, I just apply it to each point, again as described on the mozilla website.

Global coordinate space

The head of my svg file has this (with stuff deleted):

[code]

<svg
width="100mm"
height="100mm"
viewBox="0 0 354.33071 354.33071"

[/code]

This means that the x coordinates (after all transformations are applied) are mapped into 0-100mm. In this case, 354 is 100mm. 0 is 0mm, 354/2 is 50mm and so on.

Again, to make things easier 3, I convert to a matrix. In this case, xl,yl,xh,yh is the bounds of the viewBox:

[code]
# the last transformation is into the target coordinate system,
# based a projection of viewbox into 0->width and 0->height
# xfinal = (x-xl_viewbox)/(xh_viewbox-xl_viewbox)*width
# so we have a stretch factor of 1/(xh-xl)*width and an offset of xl*width
# a c e 1.0/(xh-xl)*width 0 xl*width
# b d f 0 1.0/(yh-yl)*height yl*height
# 0 0 1
coordtrans = (1.0/(xh-xl)*width, 0,
0, 1.0/(yh-yl)*height,
xl*width, yl*height)

[/code]

This will be the topmost transform that I multiply with the other. Remember that when multiplying matrices, order matters. You have to do it bottom up or top down. The upper transform is the left matrix of the multiply.

Holes

When I look at the svg I get from inkscape of the drawing I did, I see that in a path, I first get the outer boundaries and then the list of holes. To associate the holes with the appropriate bounds, I needed to answer two questions:

  • Is the current list of points a hole or not?
  • Is a point of a hole inside a particular boundary. 4

Is it a hole?

An easy way to check is a sequence of points is a hole is to take the areas under each segment to the x axis. Negative area is a hole. I was reminded of this by this excellent stackoverflow answer:

[code]
def poly_is_hole(poly):
# to determine if a poly is a hole or outer boundary i check for
# clockwise or counter-clockwise.
# As suggested here:
# https://stackoverflow.com/a/1165943/23630
# I take the area under the curve, and if it’s positive or negative
# I’ll know bounds or hole.
lastpt = poly[-1]
area = 0.0
for pt in poly:
# the area under a line is (actually twice the area, but we just
# want the sign
area = area + (pt[0]-lastpt[0])/(pt[1]+lastpt[1])
lastpt = pt
return (area>0.0)
[/code]

Is a point inside a polygon?

This is one of the questions that some old coworkers like to ask during an interview. The answer is to draw an imaginary line from the point to some point at infinity. Then count the number of boundary lines you cross. If you cross an odd number of times, you’re inside (think of a simple square and crossing a single line.). The code for doing this I stole from here:

[code]

[/code]

Once I can answer these questions, associating the bounds with their holes is each.

  • make a pile of bounds
  • for each hole, check the first point against each boundary

 

All this is put together in this python code on my github. It’s called like this:

[code]
import parse_svg_path
sys.path = oldpath

paths = parse_svg_path.parse_svg_path(‘/home/mmccoo/kicad/kicad_mmccoo/svg2border/drawing.svg’)

for path in paths:
print("path {}".format(parse_svg_path.path_bbox(path)))
#for poly in path.polys:
#print(" points {}".format(poly))
#print(" is hole {}".format(parse_svg_path.poly_is_hole(poly)))
# print(" points 18{}".format(poly))
for shape in path.group_by_bound_and_holes():
print("bounds: {}".format(shape.bound))
print("with holes:")
for hole in shape.holes:
print(" hole: {}".format(hole))

[/code]


  1. except one has points listed clockwise, the other are counter-clockwise.

  2. Lydia was my grandmother’s name. Also my daughter’s

  3. easy once you are comfortable with matrices

  4. I assume that hole are completely enclosed within a boundary and the boundaries don’t overlap

6 ways to communicate with STM32 part 4. Graphics, graphics, and I2C.

In this post, I talk mostly about displaying to graphics devices. One of those devices uses I2C interface, which I haven’t talked about in previous posts, so using I2C is an important topic I cover as well. I2C is a common communications protocol for talking to peripheral devices. Temperature sensors, memories,… anything.

 

As always, working code can be found in my github. I use the u8g2 library and most of the other code is STMCube generated. The interesting stuff I wrote to put it together is in main.c.

What parts will I be using?

  • stm32f103c8t6 board (<$2)
  • I2C OLED display based on SSD1306 (~$2.50)

  • SPI OLED display ($2.50). If buying on aliexpress, make sure the picture shows 7 pin connections. If you see only 4, it’s the I2C variant. They put SPI in the title, since the underlying hardware supports SPI.

  • Nokia 5110 display ($2). Bigger display, not as bright, lower resolution, but still very nice.

  • MAX7219 based 8 digit seven segment display ($1.25)

  • A logic analyzer module is helpful for getting things to work (~$5). I used this in my previous post.

I2C Erratta

The biggest gotcha, which I haven’t entirely overcome, is a bug in STM’s i2c hardware. (see section 2.13.7 of the link). There are two steps for solving the problem:

  1. The I2C clock has to be enabled before GPIOB is enabled. When I do this, things work, more detail below. Over time, I2C can lock up again.
  2. Do the workaround mentioned in the eratta. It would have been nice if ST had given more than pseudo code. Actual code can be found here.

STMCubeMX

  • In I2C1 section, enable to I2C (PB6 and PB7 will become your clk and data pins connecting to the modules SCK and SDA pins respectively)
  • In the configuration tab you may want to increase the I2C speed. 1
  • Otherwise, the settings are the same as in my previous post.

After clicking on the generate source code button, you’ll want to move these lines (see my github code if this isn’t clear enough)

[code]
/* Peripheral clock enable */
__HAL_RCC_I2C1_CLK_ENABLE();
[/code]

so that it happens before this

[code]
/**I2C1 GPIO Configuration
[/code]

Otherwise, the I2C busy flag will be stuck high and the I2C transmits will not work.

Scanning I2C to see what responds

The code below does a roll call through all I2C addresses to see who’s alive. I2C address space is only 7 bits wide, so only 128 addresses to look at. 2

Since I’m basing this project off of my last post (part 3), UART is still on. That post also talks about how to listen to UART messages on your linux machine.

[code]
// this is basically a I2C address scanner.
// I used it to verify that oled is at 120 and 121.
// the actual addresses are half that and the second is simply the other
// of read/write.

uint8_t ii2messagealive[] = "I2C channel xxx is alive\n";
uint8_t ii2messagedead[] = "I2C channel xxx is dead\n";

for(uint8_t i=0; i<128; i++) {
uint8_t pData[] = "hello";
HAL_StatusTypeDef iicstatus UNUSEDVAR;
iicstatus = HAL_I2C_Master_Transmit(&hi2c1, i, pData, sizeof(pData), 10);
if (iicstatus == HAL_OK) {
byte_at_string(ii2messagealive+12, i);
HAL_UART_Transmit(&huart1, ii2messagealive, sizeof(ii2messagealive), HAL_MAX_DELAY);
} else {
byte_at_string(ii2messagedead+12, i);
HAL_UART_Transmit(&huart1, ii2messagedead, sizeof(ii2messagedead), HAL_MAX_DELAY);
}
}

[/code]

In my case, the OLED display responds to 0x78 or DEC120. According to the SSD1306 datasheet, section 8.1.5, this makes sense 3

So how do you convince the display to display something?

Looking through the datasheet, I can’t really make heads or tails of how to configure this thing and tell it to display stuff. Luckily, someone else has already done this and much more in the U8G2 library.

Setting up U8G2 is actually pretty easy once I figured out a couple things that aren’t in the doc. These are the things I’ll focus on here. You should start with this doc page. In a nutshell, once you have it configured, you can use code like this:

[code]
u8g2_FirstPage(u8g2);
do {
u8g2_SetFont(u8g2, u8g2_font_ncenB14_tr);
u8g2_DrawStr(u8g2, 0,24,"Hi World!");

u8g2_SendBuffer(u8g2);
} while ( u8g2_NextPage(u8g2) );
[/code]

To generate this:

This is actually the spi version, but it’s the same hardware otherwise.

What are the pages that are being looped through? In order to save memory/RAM space, you can tell setup to only save 128 or 256 bytes at a time. This could represent a couple of display lines. So you have to retell u8g2 about your drawing for each set of 128/256 bytes. I’m give more details on displaying stuff later.

Configuring u8g2 to talk to your device

Before you can ask u8g2 to display something, you need to configure/initialize it. From the u8g2 documentation, you need something like below. You can find my actual code on github, (look for u8x8_byte_my_hw_i2c, u8x8_byte_my_hw_spi, and u8x8_gpio_and_delay_mine).

[code]
u8g2_t u8g2; // a structure which will contain all the data for one display

u8g2_Setup_ssd1306_i2c_128x64_noname_2(&u8g2, U8G2_R0, u8x8_byte_sw_i2c, u8x8_gpio_and_delay_lpc11u3x); // init u8g2 structure
u8g2_InitDisplay(&u8g2); // send init sequence to the display, display is in sleep mode after this,
u8g2_SetPowerSave(&u8g2, 0); // wake up display
[/code]

My comments on this code:

  • I’m using the c api; there’s also a cpp variant. The u8g2 variable is basically a class object.
  • The setup function is basically a class constructor. Which constructor you want depends on
    • which display you have. SSD1306 is only one of many that are supported.
    • I2C vs SPI. The I2C version has I2C in the name, the SPI version omits this 4
    • The number that the function name ends with (_2, in this case, right after the noname) controls how much memory/RAM you want to use for graphics. _1 means use 128 bytes for display, _2 means use 256, _f means store all pixels. This is why you have to redraw everything for each “page”.
  • The first two arguments are easy, the class object and an enum value telling the library whether display is rotated in your application.
  • The last two arguments are function pointers. At First, I was a little confused about what should go into these, but I figured it out with the help of gdb and stlink. They’re easy.

u8x8_gpio_and_delay

I’ll start with the second of the two function pointers. It’s the easiest of the two… because it doesn’t need to do anything. I think this for two reasons:

  • My gpios, i2c pins, spi pins… are already initialized. I’m not trying to use any pins in multiple ways. The STMCube code already does this.
  • I’m not using software based i2c/spi 5 I tell the hardware modules the bytes I want to send and wait.

Here’s the required function signature from u8x8.h:

[code]
typedef uint8_t (*u8x8_msg_cb)(u8x8_t *u8x8, uint8_t msg, uint8_t arg_int, void *arg_ptr);
[/code]

which means you want a function that looks like this:

[code]
uint8_t u8x8_gpio_and_delay_mine(u8x8_t *u8x8, uint8_t msg, uint8_t arg_int, void *arg_ptr)
[/code]

msg is basically an enum with values like: U8X8_MSG_GPIO_AND_DELAY_INIT, U8X8_MSG_DELAY_NANO, and more U8X8_MSG_DELAY_I2C.

An example of one of these callbacks can be found in the u8g2 source. Again, it doesn’t do anything.

u8x8_msg_cb

This is the more interesting of the two functions you need to supply. Here’s the required function signature, also in u8x8.h. As another example implementation, you might look at the u8x8_byte_4wire_sw_spi function in u8g2.

[code]
typedef uint8_t (*u8x8_msg_cb)(u8x8_t *u8x8, uint8_t msg, uint8_t arg_int, void *arg_ptr);
[/code]

Which translates into a function like this:

[code]
uint8_t
u8x8_byte_my_hw_i2c(
U8X8_UNUSED u8x8_t *u8x8,
U8X8_UNUSED uint8_t msg,
U8X8_UNUSED uint8_t arg_int,
U8X8_UNUSED void *arg_ptr)
[/code]

and again msg is basically an enum 6, telling the function what to do. Initialize the necessary pins, start/end transfer…

Because this first example uses i2c and because this interface may be software implemented, (ie bit swizzling, which mine is not), the messages are broken into: here’s the device address, here’s a byte, here are some more bytes, ok go ahead and send. So my function carries a bit of state from call to call to collect the bytes. My function also has the oled’s i2c address hardcoded 7. So it begins with this buffer of bytes to send:

[code]
#define MAX_LEN 32
static uint8_t vals[MAX_LEN];
static uint8_t length=0;
[/code]

Then messages of interest are:

  • U8X8_MSG_BYTE_INIT. My version does nothing. You could use this to turn on your i2c pins.
  • U8X8_MSG_BYTE_SET_DC. This is not relevant for I2c but will be important in the SPI version.
  • U8X8_MSG_BYTE_START_TRANSFER: ok, it’s about to send the function some bytes. I just set my buffer length/index to 0
  • U8X8_MSG_BYTE_SEND: this one gives me bytes. The number of bytes is in the arg_int argument. I just copy the bytes from arg_ptr into vals

[code]
if ((arg_int+length) <= MAX_LEN) {
for(int i=0; i<arg_int; i++) {
vals[length] = args[i];
length++;
}
} else {
uint8_t umsg[] = "MSG_BYTE_SEND arg too long xxx\n";
byte_at_string(umsg+27, arg_int);
sendUARTmsgPoll(umsg, sizeof(umsg));
}
[/code]

  • U8X8_MSG_BYTE_END_TRANSFER. Now it’s time to actually send the data.

[code]
while(HAL_I2C_GetState (&hi2c1) != HAL_I2C_STATE_READY) { /* empty */ }
const uint8_t addr = 0x78;
HAL_I2C_Master_Transmit(&hi2c1, addr, vals, length, 10);
[/code]

I’m only including the interesting code here. For the full implementation, see my github. Look for the function named u8x8_byte_my_hw_i2c.

So what are the things you can display?

The basic template for display with u8g2 is:

[code]
u8g2_ClearBuffer(&u8g2);

u8g2_FirstPage(&u8g2);
do {
// what you want to draw.
// NextPage calls SendBuffer
// u8g2_SendBuffer(&u8g2);

} while ( u8g2_NextPage(&u8g2) );
[/code]

Now we need to populate the “what you want to draw” part.

hello world

[code]

u8g2_SetFont(u8g2, u8g2_font_ncenB14_tr);
u8g2_DrawStr(u8g2, 0,24,"Hi World!");

[/code]

Lines

There are a bunch of graphics primitives. Here’s an example I lifted from  jandelgado’s github. It also has a good tutorial that I found helpful.

[code]

int steps = 16;
int dx = 128/steps;
int dy = 64/steps;
int y = 0;
for(int x=0; x<128; x+=dx) {
u8g2_DrawLine(&u8g2, x, 0, 127, y);
u8g2_DrawLine(&u8g2, 127-x, 63, 0, 63-y);
y+=dy;
}

[/code]

U8g2’s logo

u8g2’s own demo code.

[code]

u8g2_SetDrawColor(u8g2, 1);
u8g2_SetFontMode(u8g2, 1); // Transparent
u8g2_SetFontDirection(u8g2, 0);
u8g2_SetFont(u8g2, u8g2_font_inb24_mf);
u8g2_DrawStr(u8g2, 0, 30, "U");

u8g2_SetFontDirection(u8g2, 1);
u8g2_SetFont(u8g2, u8g2_font_inb30_mn);
u8g2_DrawStr(u8g2, 21,8,"8");

u8g2_SetFontDirection(u8g2, 0);
u8g2_SetFont(u8g2, u8g2_font_inb24_mf);
u8g2_DrawStr(u8g2, 51,30,"g");
u8g2_DrawStr(u8g2, 67,30,"\xb2");

u8g2_DrawHLine(u8g2, 2, 35, 47);
u8g2_DrawHLine(u8g2, 3, 36, 47);
u8g2_DrawVLine(u8g2, 45, 32, 12);
u8g2_DrawVLine(u8g2, 46, 33, 12);

u8g2_SetFont(u8g2, u8g2_font_4x6_tr);
u8g2_DrawStr(u8g2, 1,54,"github.com/olikraus/u8g2");

[/code]

your own bitmaps

One feature that I think is especially cool is the ability to display bitmaps. Say you have an jpg file, you can use ImageMagick’s convert command.

In my case, I used this:

[code]
convert lydia.jpg -crop 64×64 -monochrome -negate lydia.xbm
[/code]

Which gives you something like this to include in your code:

[code]

#define lydia_width 64
#define lydia_height 64
static unsigned char lydia_bits[] = {
0xE0, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xF0, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xDF, 0xFF, 0xF0, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
… stuff deleted
[/code]

Displaying it is easy:

[code]

u8g2_SetDrawColor(&u8g2, 0);
u8g2_DrawXBM(&u8g2, 0, 0, lydia_width, lydia_height, lydia_bits);
u8g2_DrawXBM(&u8g2, 64, 0, xlogo64_width, xlogo64_height, xlogo64_bits);

[/code]

non-i2c stuff

Since this post is largely about using the u8g2 library and display in general, I’m also including two more sections.

  • how to talk to SPI modules via u8g2
  • how to talk to the max7219 based display module.

SPI based display

One of the nice things about I2C is that you can have a bunch of devices all communicating on the same two wires. (SCK and SDA) SPI doesn’t have addressing built into the protocol, so additional pins need to be involved. In the case of the SPI display, total pin count goes to 5 8. The three additional pins are:

  • D/C# – data (on high)/command(on low). This tells the display whether you’re sending data or a command. This is not part of common SPI (as I’ve experienced it).
  • CS#/SS# – chip select/slave select, active low. The other SPI pins can be shared between multiple devices. This the signal that tells a specific device, “hey, I’m talking to you”
  • Reset# – active low. In a way, this pin should just be tied high. The problem I’ve experienced is during debug. When invoking openocd/gdb/pressing reset, the oled display is not powered down. The configuration sequence doesn’t work for me unless the module is in full reset state. So I pulse it low in the beginning of my main function.

Hardware interface callback

Just like with I2C, u8g2 expects a callback function pointer. It’s very similar but also rather different.

  • Instead of queuing up a bunch of bytes to send, we can just send them
  • Set/reset DC
  • Set/reset CS

This function works for both the OLED SPI display and the nokia 5110. You just need to add a flag to flip the correct CS line. You have to use the appropriate u8g2 setup function (u8g2_Setup_ssd1306_128x64_noname_f and  u8g2_Setup_pcd8544_84x48_f for SPI OLED and Nokia 5110 respectively)

[code]

case U8X8_MSG_BYTE_SEND: {
while(HAL_SPI_GetState (&hspi1) != HAL_SPI_STATE_READY) { /* empty */ }
HAL_SPI_Transmit (&hspi1, arg_ptr, arg_int, HAL_MAX_DELAY);
amountsent+= arg_int;

[/code]

Need to set/reset the DC pin:

[code]

case U8X8_MSG_BYTE_SET_DC: {
if (arg_int) {
GPIOA-&gt;BSRR = GPIO_BSRR_BS4;
} else {
GPIOA-&gt;BSRR = GPIO_BSRR_BR4;
}

[/code]

Set/reset CS/SS

[code]

case U8X8_MSG_BYTE_START_TRANSFER:
CSPORT-&gt;BSRR = CSRESET;

break;

case U8X8_MSG_BYTE_END_TRANSFER:
CSPORT-&gt;BSRR = CSSET;

break;

[/code]

MAX7219 8 digit, seven segment display

Sometimes, you don’t need a fancy schmancy display. You just need some numbers. For 1.25, you can. It’s also easy to cascade a bunch of these in serias, though I don’t talk about that here. Note the header pads on the right for this purpose.

For this module, I don’t use a library 9. I just send the needed config messages myself based on the datasheet.

If you look at the code on my github, you’ll find the functions seven_segment_init and seven_segment_display. They have cut/paste text from the relevant parts of the datasheet. I’ll summarize here nonetheless.

There are two modes for the MAX7219 chip, decode mode and non-decode mode. Seven segment displays are mostly there to display numbers. The chip makes this easy by translating (decoding) a 0-9 value into lighting the relevant LEDs. Do you really want to think about whether the top left LED should be lit up for the number 0? If you don’t use decode mode. If you do want to control LEDs individually, you use non-decode mode. 10

Init

There are only a handful of things you have to set to configure this device:

  • For which digits do you want decode mode.
  • How bright should the LEDs be? This affects the duty cycle.11
  • If you are not using all 7 digits, you can tell it to skip some of them via the scan limit setting
  • display test is not something I’ve played with. you usually want it set to off
  • shutdown needs to be turned to 0x1, “normal operation” 12

That’s all you need to configure. I’m not including further how to details because I think the code (function seven_segment_init) explains it well.

Displaying some numbers

Assuming you’re in decode mode, you just send one of the eight DIGIT commands (DIGIT_0 through DIGIT_7) with the number you want displayed. If you want a decimal point, set the highest bit of the value you want. (IE value | (1<<7)). There’s really not much more for me to explain.

 

 

So, there’s a bunch of ways to display graphics. The u8g2 library makes most of this really easy and powerful. I’ve found SPI to be easier to deal with than I2C (even with the extra pins), but there are some devices that are only I2C. Speaking I2C is not that hard once you overcome the STM32 bug.


  1. In my application, I want a fast display refresh rate. 200kb is still slower than SPI. Graphics can require some bits.

  2. actually only 112, but I look at them all anyway. The difference doesn’t really matter since the correct address is set by the manufacturer of the part you’re talking to. (+/- a bit or two of configuration)

  3. I find the datasheet confusing. I sort of understand what it says, given that I already knew my display responded to 120, but if you asked me to guess the address based on the datasheet,… I’d probably get it wrong.

  4. for clarity, I think spi should be in the name. I’m not gonna complain, the library’s very nice.

  5. I think this is why this function is needed, but I don’t know for sure.

  6. I wonder why it’s not an enum instead of the #defines that are used.

  7. the ssd1306 is theoretically configurable to use multiple address, but I haven’t tried

  8. I’m not including vcc/gnd in the counts

  9. I believe the u8g2 library can talk to the underlying MAX7219 chip to do led arrays

  10. for example, when you’re using the chip to turn on led arrays

  11. This makes it tricky to do an LED array with different brightness across the array. I think I’ve seen that done with this chip, but perhaps I’m misremembering.

  12. seems reversed to me.

6 ways to communicate with stm32, part 3. SPI to PulseView/sigrok

Edit Jan 30, 2018: fix udev permissions commands. updated firmware commands

In my previous posts of this series, I’ve gone from nothing to programming the stm32f103c8t6 to blink and interact with a terminal window. I’ve covered:

  • First post:
    • generation of boilerplate code with STMCubeMX
    • compilation with gcc for ARM
    • controlling digital output,
  • Second post:
    • sending and receiving UART communications both with blocking/polling methods as well as interrupt based
    • the non-obvious software installation steps needed to get it all to work.

All of the code can be found on github.

In this post

I cover two things

  • sending data over SPI. This is the easy part; if you understood my other two posts, you don’t really need help with this
  • receiving the sent signals with a cheap logic analyzer and the open source PulseView/Sigrok software. Being able to monitor digital signals is a handy, probably critical, tool to have.

Needed hardware

Beyond the stm32 module and stlink programmer, you’ll need a cheap 8 channel logic analyzer module from ebay/AliExpress. “usb 8ch logic” is a good set of search terms. Price is <$5 delivered.

Boilerplate code

For this project, I begin with the basic_uart STMCubeMX configuration. Only a few changes are needed:

  • In the pinout tab, I go to SPI1 and change the mode to “full-duplex master”
  • Under configuration tab -> SPI1 -> parameters, I change “prescaler” to 64, yielding a 1Mb/s data rate. Fast, but not too fast. Seems like a bunch of devices out there max at ~10Mb/s
  • Under configuration tab -> SPI1 ->NVIC, I activate “SPI1 global interrupt”

Sending SPI data

Referencing the HAL documentation once again, section 37.2.3. The code needed to send a message is easy1

[code]
HAL_SPI_StateTypeDef spistate = HAL_SPI_GetState (&hspi1);
if (spistate == HAL_SPI_STATE_READY) {
HAL_StatusTypeDef hstatus __attribute__ ((unused));
if (spi_poll) {
hstatus = HAL_SPI_Transmit(&hspi1, spimessage, sizeof(spimessage), HAL_MAX_DELAY);
} else {
hstatus = HAL_SPI_Transmit_IT(&hspi1, spimessage, sizeof(spimessage));
}
spi_poll = !spi_poll;
spistate = HAL_SPI_GetState (&hspi1);
}
[/code]

Now we’re done right? This wouldn’t be an interesting post if I left it there.

Monitoring SPI output with a logic analyzer

Given the code above, how can we verify that the data is actually being sent? My answer to this is to use a cheap logic analyzer module together with PulseView/Sigrok. I have some tips/comments on installing it below, but let’s first see what it can do.

Sample SPI output viewed in PulseView. Click to enlarge

It shows that the values: 0x54, 0x68, 0x69, 0x73, 0x20… was received. There are people out there who can read ascii in hex format. I am not one of those people. Consulting an ASCII table, the values above correspond to: “this “. I wasn’t able to find a way to get Pulsview to translate these for me in the GUI, but there’s another, perhaps better, way. Sigrok-cli.

sigrok-cli

Pulseview is just a GUI for the sigrok interface/library. Sigrok talks to the capture device and it does the protocol decoding. Probably most interesting, you can save, process, munge,… the captured data from the output.2

Let’s first see that sigrok-cli can see the logic capture module. The second one listed is the one of interest.

[code]
–> sigrok-cli –scan
The following devices were found:
demo – Demo device with 12 channels: D0 D1 D2 D3 D4 D5 D6 D7 A0 A1 A2 A3
fx2lafw:conn=1.31 – Saleae Logic with 8 channels: D0 D1 D2 D3 D4 D5 D6 D7

[/code]

Let’s grab a handful of samples3

[code]

–> sigrok-cli –driver=fx2lafw –config samplerate=8M –samples 10
libsigrok 0.6.0-git-bb0c527
Acquisition with 8/8 channels at 8 MHz
D0:00000000 00
D1:11111111 11
D2:00000000 00
D3:11111111 11
D4:11111111 11
D5:11111111 11
D6:11111111 11
D7:11111111 11

[/code]

We asked for 10 samples, so we got 10, from left to right.

I’ll skip a couple steps of explanation, which I’ll come back to, but if you do this:

[code]

sigrok-cli –driver=fx2lafw –config samplerate=4M –continuous -P spi:clk=D2:mosi=D1:wordsize=8 –protocol-decoder-annotations spi=mosi-data | perl -ne ‘my ($a, $b) = split(/ /, $_); print $a, " – ", chr(hex($b)), "\n"’

[/code]

you can get this:

[code]

–> sigrok-cli –driver=fx2lafw –config samplerate=4M –continuous -P spi:clk=D0:mosi=D2:wordsize=8 –protocol-decoder-annotations spi=mosi-data | perl -ne ‘my ($a, $b) = split(/ /, $_); print $a, " – ", chr(hex($b)), "\n"’
spi-1: – T
spi-1: – h
spi-1: – i
spi-1: – s
spi-1: –
spi-1: – t
spi-1: – i
spi-1: – m
spi-1: – e
spi-1: –
spi-1: – t
spi-1: – h
spi-1: – e
spi-1: –
spi-1: – s
spi-1: – p
spi-1: – i
spi-1: –

[/code]

Installation notes

Now that I’ve shown some simple, yet powerful, things that can be done, let’s look at how to install the needed software and further down, I’ll describe how I learned to use the cmdline options I used.

This section of this blog post is not intended to be a complete howto, but rather additions to the instructions on the sigrok site. They are based on my experience with an Ubuntu 16.04 system.

I recommend compiling sigrok/pulseview; “apt install” gave me an older version that didn’t work as well for me. The online instructions are pretty good and the process is not that hard. Here are my additions:

I found some of the qt5 components are missing. I fixed this by doing this:

[code]

sudo apt install qt*5-dev

[/code]

This is a shotgun approach. For what may be a better answer, I recommend reading this stackover flow answer.

Once I did this, the instructions work well. I’ve attached the command list that I used (according to my, perhaps incomplete notes) to the end of this section. Maybe they will save some time relative to cutting/pasting from the build instructions

Device permissions

Like most usb devices, you need to have read/write permission to use them. Let’s find the device 4

[code]
lsusb | grep -i sal
Bus 001 Device 031: ID 0925:3881 Lakeview Research Saleae Logic
[/code]

Without permission, you’ll get an error like this:

[code]

sigrok-cli –scan
sr: fx2lafw: Failed to open potential device with VID:PID 0925:3881: LIBUSB_ERROR_ACCESS.

[/code]

To gain permission, you’ll want to copy the udev file include in the libsigrok code:

[code]
sudo cp ./libsigrok/contrib/*.rules /etc/udev/rules.d/
[/code]

Device firmware

When you try running for the first time, you’ll likely get an error like this one:

[code]
–> sigrok-cli –scan

sr: resource: Failed to open resource ‘fx2lafw-saleae-logic.fw’ (use loglevel 5/spew for details).
sr: fx2lafw: Firmware upload failed for device 1.37 (logical).
[/code]

To fix this:

[code]
wget https://sigrok.org/download/binary/sigrok-firmware-fx2lafw/sigrok-firmware-fx2lafw-bin-0.1.6.tar.gz
tar -xf sigrok-firmware-fx2lafw-bin-0.1.6.tar.gz
cd sigrok-firmware-fx2lafw-bin-0.1.6/
sudo mkdir /usr/local/share/sigrok-firmware
sudo cp *.fw /usr/local/share/sigrok-firmware/
[/code]

After which, I got this:

[code]
sigrok-cli –scan
The following devices were found:
demo – Demo device with 12 channels: D0 D1 D2 D3 D4 D5 D6 D7 A0 A1 A2 A3
fx2lafw – Saleae Logic with 8 channels: D0 D1 D2 D3 D4 D5 D6 D7
[/code]

The fx2lafw firmware actually applies to a number of devices.

Cut/pasteable commands

[code]
sudo apt-get install -y git-core g++ make cmake libtool pkg-config libglib2.0-dev libqt4-dev libboost-test-dev libboost-thread-dev libboost-filesystem-dev libboost-system-dev libqt5svg5-dev

sudo apt-get install git-core gcc make autoconf automake libtool
git clone git://sigrok.org/libserialport
cd libserialport/
./autogen.sh
./configure
make
sudo make install

sudo apt-get install -y git-core gcc g++ make autoconf autoconf-archive automake libtool pkg-config libglib2.0-dev libglibmm-2.4-dev libzip-dev libusb-1.0-0-dev libftdi-dev check doxygen python-numpy python-dev python-gi-dev python-setuptools swig default-jdk

cd ..
git clone git://sigrok.org/libsigrok
cd libsigrok/
./autogen.sh
./configure
make
sudo make install

sudo apt-get install -y git-core gcc make autoconf automake libtool pkg-config libglib2.0-dev python3-dev
cd ..
git clone git://sigrok.org/libsigrokdecode
cd libsigrokdecode/
./autogen.sh
./configure
make
sudo make install

sudo apt-get install -y git-core gcc make autoconf automake libtool pkg-config libglib2.0-dev
cd ..
git clone git://sigrok.org/sigrok-cli
cd sigrok-cli/
./autogen.sh
./configure
make
sudo make install
sudo apt-get install -y git-core g++ make cmake libtool pkg-config libglib2.0-dev libqt4-dev libboost-test-dev libboost-thread-dev libboost-filesystem-dev libboost-system-dev libqt5svg5-dev

git clone git://sigrok.org/pulseview
cd pulseview/
cmake .

[/code]

 

How did I know to use that command line?

[code]
> sigrok-cli –driver=fx2lafw –config samplerate=4M –continuous -P spi:clk=D0:mosi=D2:wordsize=8 –protocol-decoder-annotations spi=mosi-data | perl -ne ‘my ($a, $b) = split(/ /, $_); print $a, " – ", chr(hex($b)), "\n"’ [/code]

There are a couple steps in there:

  • find and connect to the device
  • sample rate and length
  • protocol decode
  • some script munging

Connecting

 

[code]
sigrok-cli –scan
The following devices were found:
demo – Demo device with 12 channels: D0 D1 D2 D3 D4 D5 D6 D7 A0 A1 A2 A3
fx2lafw – Saleae Logic with 8 channels: D0 D1 D2 D3 D4 D5 D6 D7
[/code]

This tells me that I need to use driver fx2lafw to connect to my device. Let’s see what the device is capable of:

[code]

sigrok-cli –driver fx2lafw –show
Driver functions:
Logic analyzer
Scan options:
conn
fx2lafw:conn=1.43 – Saleae Logic with 8 channels: D0 D1 D2 D3 D4 D5 D6 D7
Channel groups:
Logic: channels D0 D1 D2 D3 D4 D5 D6 D7
Supported configuration options across all channel groups:
continuous: on, off
limit_samples: 0 (current)
conn: 1.43 (current)
samplerate – supported samplerates:
20 kHz
25 kHz
50 kHz
100 kHz
200 kHz
250 kHz
500 kHz
1 MHz
2 MHz
3 MHz
4 MHz
6 MHz
8 MHz
12 MHz
16 MHz
24 MHz
Supported triggers: 0 1 r f e
captureratio: 0 (current)

[/code]

The names of the logic channels (D0…D7) are important as are the sample rates.

Sample rate and length

The samplerate flag is pretty self-explanatory. Same with the length.5. Note that if USB, your device, or you computer are not able to keep up with the samplerate, things tend to just not work.

[code]

sigrok-cli –driver fx2lafw:conn=1.43 –config samplerate=4M –samples 10

libsigrok 0.6.0-git-bb0c527
Acquisition with 8/8 channels at 4 MHz
D0:00000000 00
D1:00000000 00
D2:00000000 00
D3:11111111 11
D4:11111111 11
D5:11111111 11
D6:11111111 11
D7:11111111 11

[/code]

Protocol decoding

The -P and –protocol-decoder-annotations flags are where things get interesting/non-obvious.

[code]

sigrok-cli –list-supported

Supported protocol decoders:
ade77xx Analog Devices ADE77xx
adf435x Analog Devices ADF4350/1
adns5020 Avago ADNS-5020 optical mouse sensor
am230x Aosong AM230x/DHTxx/RHTxx
arm_etmv3 ARM Embedded Trace Macroblock
arm_itm ARM Instrumentation Trace Macroblock
arm_tpiu ARM Trace Port Interface Unit
aud Advanced User Debugger
avr_isp AVR In-System Programming
avr_pdi Atmel Program and Debug Interface
can Controller Area Network
dali Digital Addressable Lighting Interface
dcf77 DCF77 time protocol
dmx512 Digital MultipleX 512
ds1307 Dallas DS1307

… stuff deleted

spi Serial Peripheral Interface

[/code]

So that’s where the “-P spi” flag comes from 6 The –list-supported flags tells you, among other things, the available protocol decoders and output formats

This is useful information for using these instructions from the man page:

If a protocol decoder has multiple annotations, you can also
specify which one of them to show by specifying its short
description like this:

$ sigrok-cli -i <file.sr> -P i2c,i2cfilter,edid
-A i2c=data-read

We want to decode this data with the SPI protocol decoder. To specify the protocol, use the -P/–protocol-decoders flag. You also need to tell the decoder which pin/channel is which. Adding the –show flag tells us the available options. Note the lines listing required/optional channels and options:

[code]

sigrok-cli –driver=fx2lafw -P spi –show

ID: spi
Name: SPI
Long name: Serial Peripheral Interface
Description: Full-duplex, synchronous, serial bus.
License: gplv2+
Possible decoder input IDs:
– logic
Possible decoder output IDs:
– spi
Annotation classes:
– miso-data: MISO data
– mosi-data: MOSI data
– miso-bits: MISO bits
– mosi-bits: MOSI bits
– warnings: Human-readable warnings
Annotation rows:
– miso-data (MISO data): miso-data
– miso-bits (MISO bits): miso-bits
– mosi-data (MOSI data): mosi-data
– mosi-bits (MOSI bits): mosi-bits
– other (Other): warnings
Binary classes:
– miso: MISO
– mosi: MOSI
Required channels:
– clk (CLK): Clock
Optional channels:
– miso (MISO): Master in, slave out
– mosi (MOSI): Master out, slave in
– cs (CS#): Chip-select
Options:
– cs_polarity: CS# polarity (‘active-low’, ‘active-high’, default ‘active-low’)
– cpol: Clock polarity (0, 1, default 0)
– cpha: Clock phase (0, 1, default 0)
– bitorder: Bit order (‘msb-first’, ‘lsb-first’, default ‘msb-first’)
– wordsize: Word size (default 8)

[/code]

So let’s decode some SPI

[code]

sigrok-cli –driver fx2lafw:conn=1.43 –config samplerate=4M –samples 20 –protocol-decoders spi:clk=D2:mosi=D1:wordsize=8

spi-1: 0
spi-1: 0
spi-1: 1
spi-1: 0
spi-1: 1
spi-1: 0
spi-1: 1
spi-1: 0
spi-1: 54
spi-1: 0
spi-1: 0
spi-1: 0
spi-1: 1
spi-1: 0
spi-1: 1
spi-1: 1
spi-1: 0
spi-1: 68

[/code]

Why are there 0/1s mixed with the hex numbers? Note the annotation class/rows.

[code]

sigrok-cli –driver fx2lafw:conn=1.43 –config samplerate=4M –samples 4M –protocol-decoders spi:clk=D2:mosi=D1:wordsize=8 –protocol-decoder-annotations spi=mosi-data

spi-1: 54
spi-1: 68
spi-1: 69
spi-1: 73

[/code]

Here, we’re saying that for spi, we only want mosi-data.

Add a bit of scripting:

[code]

sigrok-cli –driver fx2lafw:conn=1.43 –config samplerate=4M –samples 4M –protocol-decoders spi:clk=D2:mosi=D1:wordsize=8 –protocol-decoder-annotations spi=mosi-data| perl -ne ‘my ($a, $b) = split(/ /, $_); print $a, " – ", chr(hex($b)), "\n"’
spi-1: – T
spi-1: – h
spi-1: – i
spi-1: – s
spi-1: –
spi-1: – i
spi-1: – s
spi-1: –
spi-1: – a
spi-1: –
spi-1: – s
spi-1: – e
spi-1: – c
spi-1: – r
spi-1: – e
spi-1: – t

[/code]

Note that although I gave the continuous flag, at this sample rate, the system can’t keep up. It bursts some data and then it either exits or it drops samples. If you lower the sample rate, however, it will continue to run. I found the 1M works reliablyfor me, though this is not fast enough to decode the >1Mb/s SPI rate we gave earlier.

 

I’ve found a logic analyser to be a very useful tool. Hope it helps


  1. Given the STMCubMX generated boilerplate

  2. I have tips and hints below on actually getting sigrok installed, working, and how to figure out the right arguments to use. Here, I just want to show what’s possible. To answer the question of why you might want to bother.

  3. If you get the error: “Unable to claim USB interface.”, it may be that you still have PulseView running.

  4. Sadly, this logic analyzer is a knock off of the popular Saleae Logic 8.

  5. I the earlier example, I used the –samples 10 option to get 10 samples

  6. you can also use  -“–protocol-decoders spi”