The good part of this release is the availability on next Ubuntu Maverick. If you don’t know, Ubuntu Maverick will be multitouch. Before making the announcement public, we’ve joined the HCI group, and they have done documentation about Multitouch in Ubuntu, and PyMT in Ubuntu.
That’s a very good news, and i hope that we’ll be able to enhance our collaboration with Ubuntu, and share experience.

So, what’s next ?

Since people are asking always that “we should do a roadmap”, PyMT Roadmap is updated.
Next version will be the PyMT 0.5.1, and release is scheduled for the 6 September. This will include another set of fixes, found during the 0.5 release. And this will be the version shipped on next Ubuntu too. Auto detection of MT Hardware, fixes text rendering outside of the text input, gstreamer appsink missing, more unit test…

Starting September, the “PyMT Week Report” will back. Since the community is growing, it will be a good thing to know what’s new on the toolkit every weeks, and explain if we are hitting trouble, or if all the developments is ok

Then… I’ve lot of idea on my mind, and will focus on :

• Spatialization: be able to handle 1 millions of widgets, added on an infinite plane.
• Graphics: finish and use graphics library (new from 0.5) for all of our widgets.
• App subsystem: replace the old MTPlugin with a new Application subsystem, that permit the use to browse and launch any PyMT application available on his system
• Enhance CSS writing (less or sass-lang)

Then in november, i’ll try to port PyMT on Javascript… using Pyjamas and HTML5.

PyMT is really going to be… awesome !

We’ve been working hard to make this reality, and many a new feature has been added and quite a few bugs have been squashed. I suggest you read the full changelog and, if you already have a PyMT 0.4 application, also the migration guide.

One of the coolest new things with this release is the availability of portable binary packages for Windows and OS X. Those come bundled with everything you need (on Windows, even Python) to get started. You simply download the package for your platform and run it. We didn’t provide a portable package for Ubuntu, but it’s ridiculously easy to install PyMT there anyways. On Ubuntu 10.10, all you need is:

sudo apt-get install python-pymt

PyMT has native support for multitouch devices on Linux that are supported by the kernel, all Windows 7 multitouch devices, all of Apple’s multitouch accessories and much more. If you know basic python, PyMT is the easiest way to create multitouch applications or to just test your hardware.

In future releases we’re planning to fully use a rewrite of our current OpenGL abstraction and other performance-critical parts (that we start doing in C) that will allow for much higher application speed, less battery consumption and OpenGL ES/3.0 compatibility so that we can smoothly run on portable slate/pad devices.

Lastly, see what people have done with it (planet readers, click the images to get to the videos):

I hope that sparked your interest. We also hope you enjoy PyMT. If there are any questions, bugs, problems or feature requests, let us know. There’s a mailing list, a google code issue tracker and our IRC channel at irc.freenode.net in #pymt.

• I added a new spelling provider to PyMT that abstracts from individual spellchecking libraries. That means you can use your favorite spellchecking library, which is important considering that PyMT is cross-platform.
• I added two actual spelling providers that implement this protocol: One enchant spelling provider (usable after installing enchant) and one provider using OS X’s native AppKit spellcheckers (so you get that out of the box on OS X).
• Mathieu once wrote a basic virtual keyboard with spelling suggestions which I adapted, cleaned and merged.
• PyMT obviously already had some text input widgets, which I improved (e.g. MTTextArea).
• I began working on a version of MTTextInput with added spellchecking (like OO.org with red lines drawn for incorrectly spelled words), but that needs some more love.
• One of the more concrete objectives of my task was a Swype-like keyboard for PyMT. I created a prototype for that, see the video below.
• Another concrete objective was a split keyboard (split into two parts, one half for the left, one for the right hand) that adjusts to your hand’s properties (e.g. size). To achieve this, a substantial amount of changes was needed to our vision tracking application (Movid):
• For the keyboard to adjust to the user’s hands, a handtracking algorithm was needed that I implemented for Movid. It detects the fingertips of the hand as well as the hand’s center. These are just seen as a certain type of ‘blobs’ internally.
• These blobs need to be tracked over a sequence of frames from the camera. Additionally, we also want to find simple touches (without all the hand information). For that, I added and integrated BlobFinder and BlobTracker modules that obey a common format so they’re easily interchangable.
• When your camera senses a blob on the touch surface, the application needs to perform a mapping to get the blob into screen coordinates. We do that using a calibration module, which I had started before SoC. I finished it and merged it back into our master branch.
• As an extra feature, I added a PyMT module that you can use to calibrate your tracker from within your client application, eliminating the need to switch applications. I also added a Flash GUI for the calibration so that you can easily do it on any remote computer via our web interface.
• To actually send the handtracking to the client application, Mathieu added a TUIO2 module to Movid. I started a PyMT input provider for TUIO2. Both of which is work in progress, but I believe we’re the first project to adapt TUIO2 (there’s not even a reference implementation yet).
• The result of that can be seen in the second video below. Also, make sure to read the vimeo description.
• Other than that we now also provide portable binary packages for PyMT 0.5 for both Windows and OSX. I created the OSX package, so it’s no longer a major pain to install. You just download and run it.
• And, of course, many more fixes!

Some of that is already in PyMT 0.5. All of the Movid stuff will be in the first release. In future releases we shall see much improved versions of these prototypes and hopefully even context aware word suggestions.

Here are the two promised videos, if you’re reading this through a planet, please go directly to my blog.

Prototype WipeToType Keyboard for PyMT from Christopher Denter on Vimeo.

Ergonomic multitouch keyboard prototype from Christopher Denter on Vimeo.

Thanks to all the people who made this possible. Thanks Google, Christian, Pawel, Mathieu and Thomas, for being (a) fantastic mentor(s). It has been a great pleasure and privilege to work with you in GSoC 2010 and I sure will continue to work on both projects.

I’m writing this blog post just to remember what i’ve done, and how. Don’t want to forget it for the next time It’s incomplete, and done for developer.

Last week, i’ve got a Joojoo, the old named “Crunchpad”. The hardware look good: Intel ATOM N270 + NVidia ION (GeForce 9400M) + 4Go SSD + 1Go Ram + 1.3M Webcam + USB + eGalax dualtouch screen + Wifi. At least, it look like a good hardware to hack on. Before you ask, i’ve buyed this tablet cause of NVidia ION, it support OpenGL 3.0, enough to run PyMT ! I’ll not say anything about the default OS installed on Joojoo… but will start directly how to install custom OS on it.

How to install custom OS on JooJoo

1. You can use any usb device on the joojoo (CD or USB). I’ve used Unetbootin to create a custom USB boot key for Lucid Ubuntu. Don’t select net install, wifi driver is not included in the setup.
2. Plug a usb switch + usb keyboard + usb key on the usb port of Joojoo
3. Stay pressed on the DEL key + press twice the power button (once to make Joojoo logo appear, and a second longer to power the screen)
4. In the boot menu, change the order of Joojoo SSD and your USB key, to make your USB key primary.
5. Reboot by pressing twice the power button as described.

If you see nothing on the screen, reset the joojoo with the tool, and press the power button twice again.
Twice is really important, without that, the screen will stay black. On the second time, you must see the screen power up.

About drivers :

• The default wifi driver is not working, so go on http://www.realtek.com/downloads/ , search for 8192SE, and take RTL8192SE drivers for linux 2.6.
• For Nvidia ION, just download latest drivers from http://nvidia.com/.
• For touchscreen… that’s the whole problem. It’s not working by default. You can use the constructor driver available http://home.eeti.com.tw/web20/eGalaxTouchDriver/linuxDriver.htm, but dual touch is not working.

So, i’ve buyed this tablet for Dual Touch + OpenGL 3.0… Let’s check if windows is working with dual touch.

How to install window 7 on JooJoo

On this part, i would like to thank a lot Fabien for his experience on this domain.
The problem is, with 4Go SSD, you can’t install Normal Windows Seven. You must build a custom Window. To do that :

1. go on Microsoft Windows 7 Embedded
2. download the Standard 7 Toolkit
3. launch the image configurator
4. select all the things you want on windows… Like Window desktop, USB Boot, Power Management… I’ll upload my “answer” file tomorrow.
5. your image must be < 2Go.
6. burn your answer on usb (Tool menu). You’ll have a USB boot key ready to install Window Seven Embedded !

Then, it’s the same step as before, plug, reboot, install.

After installation of eGalax windows driver, the touchscreen work with dual touch !

How to capture USB on windows seven ?

Since wireshark is not able to capture USB, and no free toolkit is able to do it at this moment, i’ve used HHD Usb monitor, in trial version. Just run it, select Raw capture, Start, play with the screen, reboot the screen, play again with the screen (1 finger, then 2). And export the whole capture into HTML.
And here is the Joojoo eGalax USB Capture.

And that’s a bingo. We have bulk message with a 0×6 len, and other with 0xc len (double of 0×6.) Look really like we got 2 touch ! I don’t know what is the setup message, but look like theses messages activate the screen ! Now… go back on linux to test

Check eGalax dual touch support with python-usb

Enable usb_debug (optionnal)

Ok, this was really easy, cause of the old experience of Hack of Dell SX2210T screen. Even if the experience was not a success, learning how USB work was really funny. First thing, activate USB sniffing to check if we are doing it right :


$ sudo modprobe usb_debug
$ sudo cat /sys/kernel/debug/usb/usbmon/0u (0u is where my device is connected)


You’ll have a bunch of message, as soon as you play with the device. You can read usbmon/usb_debug documentation about the format of the message.

Search your device with python-usb

When you’re searching for device on dmesg, you’ll have :

generic-usb 0003:0EEF:720C.0001: input,hiddev96,hidraw0: USB HID v2.10 Pointer [eGalax Inc. USB TouchController] on usb-0000:00:06.0-1/input0


You can also use lsusb :

Bus 004 Device 002: ID 0eef:720c D-WAV Scientific Co., Ltd


The 0EEF is your vendor ID, and 720C is your product ID. Let’s search our device in USB busses:

def get_egalax_device(vendor=0x0eef, product=0x720c):
    busses = usb.busses()
    for bus in busses:
        for device in bus.devices:
            if device.idVendor != vendor:
                continue
            if device.idProduct != product:
                continue
            return device
    return None

Send SETUP configuration

To understand more how USB work, you can check USB nut shell, the first part is talking about Setup packet.
This is exactly what we want, since on the capture we have :


000006: Control Transfer (UP), 02.08.2010 01:50:04.275 +0.0. Status: 0x00000000
Pipe Handle: 0x855dbe94

Recipient: Device
Request Type: Vendor
Direction: Host->Device
Request: 0x0 (Unknown)
Value: 0x0
Index: 0x0
Length: 0x6


We have exactly 2 setup packet to transfer to the device. The most complicated is to understand python-usb.

• Recipient -> usb.RECIP_DEVICE
• Request Type -> usb.TYPE_VENDOR
• Direction -> usb.ENDPOINT_OUT

So, let’s send the first setup packet. This packet activate the device to talk with us (we’re receiving lot of packet from him).

conf = '\x40\x00\x00\x00\x00\x00\x06\x00'
reqType = usb.ENDPOINT_OUT | usb.TYPE_VENDOR | usb.RECIP_DEVICE
handle.controlMsg(reqType, usb.REQ_SET_CONFIGURATION, conf)

And the second request activate dual touch support (without him, we’re receiving only one touch) :

conf = '\x0a\x01\x41\x0a\x01\x41'
handle.controlMsg(reqType, usb.REQ_SET_CONFIGURATION, conf)

And, that’s done ! When we read the device, we are receiving packet !
The format is :

• Flags : 8 bits (161 = ALIVE, 160 = UP)
• X : 16 bits
• Y : 16 bits
• ID : 8 bits (65 = touch 1, 66 = touch 2)

The flags and id is not really flags/id… because the value is not common. But it’s not changing on pressure, and not in the time… Dunno why.

Still, here is the full source code of the eGalax Joojoo dual touchscreen with python-usb.

And after ?

Normally, 2.6.35 include eGalax driver, but it’s not working. Will check another day why the kernel driver don’t support dual touch for this screen !

/me smile

Medical Multitouch from Christopher Denter on Vimeo.

Reading through a planet? Click here!

For more information, see the video description. PS: Although it supports all platforms, it currently runs on ubuntu. :-)

Let me know what you think!

PyMT Spelling

Starting with PyMT 0.5 (to be released in August), we added support for spelling correction and word suggestions. This is based on my GSoC work. The code has been polished and integrated into the master branch, which will soon lead to the 0.5 release. I’m also currently working on a text input widget that indicates incorrectly spelled words as you type (just as OpenOffice would). This much is working. In a next step, I plan to add a feature that lets you select from a list of suggestions for the word that you just tapped.

WipeToType

One of the tasks of my GSoC proposal is the implementation of a Swype-like keyboard. What this means is that you just wipe over the keys that make up the word you want to type and it automatically determines which word you intended to enter. It is clearly far beyond the scope of a single, multitouch-oriented GSoC proposal to implement something as clever as a Swype clone (especially since this also requires A LOT of backend code for the intelligence). However, something remotely similar and usable should be doable and is what I’m looking for. A while back I started something like this and quickly sketched a modified version of the virtual keyboard.

This still needs much more love, but keeping in mind that I did this in a really short amount of time I think I can say that we’re getting somewhere.

FITG in Lille

Thomas, Mathieu and I have had the chance to meet in Lille at the FITG conference and present PyMT in a talk and several workshops. This was a great opportunity and in fact, it was the biggest real-life meeting of core developers and users so far.

The conference itself was a great success, both for the organizers and us. We had many people come to us and ask questions concerning PyMT and Movid. After our talk (which I think was well-received) we decided to give an additional workshop so that people interested could play with PyMT and get help from us. The room was pretty crowded and people were standing. The workshop presented a basic PyMT overview and first steps in a ‘hello world’ fashion (At least I think that’s what Mathieu was talking about. He spoke french and Thomas and I were answering questions or translating things in English). In the evening we gathered all the people that were still at the conference and went to a nice little restaurant to chat.

The next day we gave two more workshops. The idea to do the first one came up while we had breakfast. We decided to implement a simple version of the game at linerider.com with PyMT. When we arrived an hour later, we just picked a python 2D physics library that was easy to install and started live and from scratch, without any actual code having been written (or even thought about) beforehand. Luckily it all turned out well. After almost exactly one hour (in which Mathieu helped people in the audience, Thomas pointed to and explaining stuff at the projection and I coded and talked) we had finished what we were looking for in just 60 lines of unoptimized python code. The last workshop was about advanced OpenGL. Mathieu presented some of his insights that he had gathered while optimizing PyMT’s performance (great advances have been done here, by the way).

I stayed four days in total and it was absolutely worth it. Lille is a wonderful city and the conference was fantastic. The venue itself was just mind-blowing to begin with. We had a lot of fun together and obviously worked on PyMT as well. It’s even more fun if we’re in the same room! Sincere thanks to everyone involved in making these days as awesome as they were!

Reworked TextArea

For our talk in Lille, we used a very nice presentation tool (PreseMT) which is, obviously, written with PyMT. While using it for my own bachelor’s colloquium (I’m officially a Bachelor of Science now, by the way) I noticed that entering text suffered from severe limitations of the text input widget. Given that I had no multitouch-capable device around to enter text, I did it all with my hardware keyboard. I added to PyMT’s TextArea widget the ability to resize automatically depending on the text that was entered (which is what you want in PreseMT). Furthermore, the widget now properly reacts to several special keys like the arrow keys, home, end, del, pgup and pgdown.

Portable PyMT

It is no secret that installing PyMT on OSX is a major pain. This is not our fault, though. The problem simply is that installing almost anything in non-app format involves a non-trivial compilation process using MacPorts and the like. Unfortunately, one of our dependencies (gstreamer) is not easily installed this way.

Since we really don’t want our users to go through all of this, we decided to distribute portable versions of PyMT for OSX, Linux and Windows. I did the OSX version and hope to be able to finish it soon so that it can be reviewed. With it, you just download a zip file, unzip it and go. It contains everything that is needed to run PyMT.

In the course of this, I also fixed the compilation of our OpenGL-dependant cython modules for OSX.

Conclusion

Hopefully you will see a wonderful PyMT release next month. We’ve added many new features, improvements and fixed a lot of bugs. Some of my GSoC work will also go into it. In terms of GSoC, I will finish the aforementioned spelling-aware text input widget. I also intend to improve the quality of the results of the WipeToType keyboard and implement the things left on my GSoC proposal.

We’ve done one conference, using the “presemt” application. For all who wanted to use our presentation application, you can get it on PyMT Apps repository. The slides will be available in PDF soon. A lot of peoples was interested, and impressed by PyMT itself. I’ve been very glad to meet some PyMT users, specially Jay !

On the barcamp, we’ve animated 3 sessions:

1. Write your first PyMT Application (mathieu)
2. Make a game with PyMT + PyMunk (christopher + thomas)
3. Advanced OpenGL optimization + Python optimization

(Content will be available soon)

During the event, some work have been done :

• finish blobtracker in Movid
• ability to record / save PyMT events (event-recording branch)
• ability to share event over network (event-share branch)
• using PyMT + Ogre 3D with Anthony Martinet

And we’ve think a lot about the future of PyMT, and where we want to go for 1.0. More information soon

Also, a intern in the MINT team is actually researched to work on a project with PyMT. If any people is interested, don’t hesitate to send our CV to Nicolas.

It look like the future of PyMT is very good, and it’s just a start….

Now, they are doing a event on 3 days… and they are more.

Thomas Hansen and Christopher Denter from PyMT/Movid will be here ! For the very first time, our core team will be at the same place, IRL. That’s will change a lot from IRC

Be ready for our workshop, we are scheduled in Friday, 13h30, for 15 minutes + 5 minutes for discussion.
We have some ideas about how we can be really wonderful during only 15 little minutes

I’ll animate a session around coding with PyMT in Saturday. If you want to join, be ready with your laptop. It would be better if you already have PyMT installed.

For more information, you can read the FITG2010 website !