A blog

linux, python, java and movies

Menu Close

MXNet – Image iteration using mx.io.ImageRecordIter

In one of the previous posts I’ve described how to create RecordIO data set for MXNet.

Now I wanted to share an example how to iterate over the RecordIO data set. This example in many ways similar to the how to tutorial that are available on the mxnet github page. But nonetheless, I’ve decided it could be a logical continuation of a previous post and could probably help someone.

When we have a RecordIO set of images, we can use mx.io.ImageRecordIter to load and decode the data.

If you have prepared your image training set using im2rec.py correctly, you will see 4 little pictures plotted on the graph.

MXNet im2rec.py

MXNet provides a Python utility to create RecordIO packages of data that are supported by framework’s data iterators.

https://github.com/dmlc/mxnet/blob/master/tools/im2rec.py

In order to create a rec file, you first need to make a list of files using this command:

After you get .lst files, lets create a record set.

Record objects will be about the same size as the data that being used for the objects.

here we eliminating –list argument so the script will be working on .rec files.

At the end of its execution, it should display something like

and in the folder where you have your dataset you will see files

and same for train and validation sets.

 

Enjoy!

Installing MXNet deep learning framework on Windows 10

Hello, hello.

I wanted to share a quick how-to install MXNet on Windows 10 64-bit.

The solution here is a bit opinionated as I am using Anaconda to manage Python environments.

I will start from the beginning – Anaconda installation.

We will need Anaconda 4.3. You should use 64-bit Anaconda/Python 2.7 combination, as the environment we will create will be based on Python 2.7 64-bit in order to support MXNet. You can get anaconda here:

https://www.continuum.io/downloads

Follow the installation instructions and once it is done, lets go to the windows command line and type this:

having ‘anaconda’ argument in there will ensure we have common data analytics libraries, that are not necessary for MXNet, but nice to have. Also, we want to use Python 2.7 in this case, as MXNet has compatibility issues with Python 3.

Once the environment creation is complete, activate it by executing:

It is all for the terminal so far.

Lets get pre-combiled MXNet dlls!

You will need to download several packages from

https://github.com/yajiedesign/mxnet/releases

First, get the base VC14 ( Visual Studio runtime 2015) package, it doesn’t contain any specific binaries, but it provides 3rd party libraries and helps to set all the environment variables that are necessary to run MXNet.

Once you download VC14 base archive, extract its contents somewhere. A good folder candidate would be ‘mxnet’ in the root of C: or any other drive. Let’s imagine that you have created ‘D:\mxnet’ folder.

Open terminal in that folder an run

It should finish quickly. You can verify that it was successful by going into the System -> Advanced System Settings and making sure it has MXNET_HOME:”D:\mxnet” environment variable setup.

Now, lets download GPU or CPU version of precompiled MXNet from the same page where you got the base package.

The archive will have contents that you need to extract to “D:\mxnet”, it might overwrite some of the folders, which is ok.

Optional STEP:

Now, if you used GPU version of MXNet, the last part would be is to get the cuDNN nVidia library for windows, you will have to register to get it, but essentially it is free.

cuDNN is available here:

https://developer.nvidia.com/rdp/cudnn-download

Once you download the archive, extract its contents to “D:\mxnet\3rdparty\cudnn”.

END OF Optional STEP.

It is time now to try to install python binding, so you can import mxnet inside a python project.

Go to the terminal where you had your MXNet Anaconda environment activated and jump to the location: “D:\mxnet\python”. Once you are in the correct location, run

If it ran without any errors, then we are ready to do our final test. Go back to the terminal where you acivated mxnet environment and first, start Python interpreter, and then run

You should be able to import the library.

Enjoy!

 

Javascript – Update all packages in package.json

If you are trying to keep up with Angular2 updates and want to get latest libraries’ versions in your package.json:

 

Enjoy!

Python NLTK module and its download() function

I am trying to build a small machine learning service that would use Python NLTK library. As it is a pet project, and a very small one – I’ve decided to use Heroku for the hosting. And as I am using NLTK library I needed to download models and corpora by calling nltk.download() method to parse punctuation and have some other textual tricks.

Heroku doesn’t allow that method to execute, as it requires GUI interaction.

Initially I got a bit lost, as there is a way to bypass GUI when you provide specific list of NLTK corpora to download – but for certain missing models the error message provided a name of a model that was not compatible with nltl.download().

Luckily, I’ve found the page with a list of all available corpora and associated download IDs.

http://www.nltk.org/nltk_data/

Enjoy.

RE7 – Biohazard

I want to congratulate all the fans of the series. Finally there is an episode that has an old house.

An episode with puzzles in an old house. Also, an episode where you have to save bullets when you trying to kill these very, very scary mutations and family members.

I am loving the gameplay, but can not play it alone, I get scared. A true horror game as it should be. I am playing with my mom, who is about to finish the game on her own. We both are fascinated with the visuals of the game that are pushing the boundaries of what our old PS4 could deliver. Truly beautiful and scary.

I am happy there will be new episodes coming in later as DLCs. Not a big fan of the DLC concept, but I will make an exception.

A strong return in my opinion.

Ideology

Ideology – is a system where other people can wreck your life.

https://en.wikipedia.org/wiki/Ernst_Kolman

Generators in Python!

Python generators code tutorial:

UUID3 cross compatible in Java and Python

Sometimes, in the world of microservices you want to generate consistent hashes across several languages. I’ve faced a problem of getting the same UUID in Java and Python, and wanted to share how it can be done.

Java

Python

Enjoy!

PyMongo insert_many with overwrite

I’ve recently needed to insert lots of objects into the Mongo collection. The only problem was that some of the objects would have an ‘_id’ key pre-set and would conflict with existing objects in the database.

PyMongo inser_many operation doesn’t support it out of the box, so here is a work-around using bulk api:

© 2017 A blog. All rights reserved.

Theme by Anders Norén.