Category Archives: open source

Continuing work on open source

Since pointer lock is now in review, I’m sure there’ll be stuff to fix when review on the patches and tests are done, I went looking for a couple of bugs to do. The first bug relates to an updated macro. I don’t think it’ll take too long, it’s not a simple find and replace procedure as noted on the bug report. I’ve started by using mxr  to find out how many files are affected and how many instances. It shouldn’t be too much work, and I’m with a bit of discussion with Jeff Walden, I’m sure I can have it done by Thursday. (There are 214 instances in 78 files)

My second bug is something a bit more abstract. I’m thinking this bug might take more than a few weeks to complete, but I’m sure I’ll be able to get it done by reading week (crosses fingers). The biggest obstacle for me is that I don’t really have any idea where I should begin (aside from looking at the old patch shown in bug 595451). I’ve emailed Mounir Lamouri to see where I should begin, this should be an interesting adventure. =)


Pointer Lock – Developing on Windows

There’s been a lack of posts I must admit, much of it due to the lack of any real significant breakthroughs. Much of the progress has been minor ones which didn’t feel like they merited a post in itself. So now that a week has essentially come, a round up post is in order. As the title of the post suggests, much of this post will detail my experience while working on Windows.

Work Done

There were three main things that were necessary to get the tests done (assuming that the tests from last semester were all pretty much perfect). The first was that the tests needed to be updated to use the proper syntax. This meant that lines that said var pointer = navigator.pointer got changed to var pointer = navigator.mozPointer. The second thing that needed to be done was to trim unnecessary tests. Several tests from last semester had tests that overlapped with each other, and in general, if it’s tested once and it works, don’t bother testing it again, it’s a waste of time. Lastly, one thing that was noticed by a few of the test developers from last semester were that there were leaks that were being caught by mochitest.

This last task was the most time consuming of them all and it ended up also being mostly a wasted effort. Much of the leaks actually occurred in the now defunct file_MouseEvents.html. As mentioned in my previous post, when I first found this, I was extremely exciting. It took ages to find this out. I had to cut files from the test harness one by one to find which file did it. Afterwards I had to lob of entire sections of the test to finally find the culprit, and then promptly learned that it was a Windows specific issue. Luckily, it turns out that we didn’t need to be that specific, which meant we could avoid the leak totally. By specific I mean that for Firefox, the route a mousedown event takes is for the most part is the same regardless of which button is pressed. So instead of sending a mousedown event for left, middle, and right mouse button, you can send a generic one and know that it will cover all three.

The next major work was to integrate into file_movementXY.html a test to check to make sure the mouse can theoretically move beyond the element in point lock. In other words, once in pointer lock, the mouse should be able to continue to move in any direction and not worry about hitting the “edge” of the browser which would, in normal circumstances, cause the mouse to stop reporting events to the browser. Original attempts were fairly straight forward. The test would move the mouse, store the mozMovementX and mozMovementY move it almost half the screen 3 times, and then check to make sure the values added up to be larger than the screen width or height (depending on if we’re comparing X or Y). On linux it flat out failed. Instead of getting a positive number, it seemed that we were getting the negative equivalent to what we were expecting. On Windows, it was worse. For the most part, running it would give the tests a pass (which was why I pushed to my repository for peer review), but if you were to move your mouse outside of the browser when the test started, the tests would fail and something similar to the linux error occurs… a negative number. It turns out, the test could be simplified and after a chat with Diogo the current method was developed. The idea was that by moving the mouse to the center and initializing pointer lock, we know that after each move we go back to the center of the screen. We would move a quarter of the screen each time, and when the moves are finished we can declare that it works. The reason is because if we try to use synthesizeMouse and send the mouse to a location beyond the browser, the test would hang and thus the test would fail.

More recently, work was being done on file_cursorPosEvents.html. This really highlighted a lack of understanding on my part regarding asynchronous calls and events. I’m too used to coding in C and C++ with synchronous calls and often times I make the incorrect assumption that certain things are synchronous when they’re not. In such cases, we’re looking at possible random fails. Another poor habit includes not using a bool when all I use a variable for is to track if something happened or not. With this test, I’ve also found that working running pointer lock tests on Windows is wonky at best…

The Rant

I’m sure we need to test these on Windows, however, so far it seems that whatever runs on Linux works on Windows. More interestingly is that, whatever doesn’t work Linux, still has a chance to work “as expected” on Windows. As I showed Diogo on Tuesday, while movementXY would fail on linux, and it would succeed with no problems on Windows. This has caused me and my colleagues a lot of headaches (namely because I push it up to my repo for review because it works for me… just not for them). However, to really highlight the “what the heck”-ness of developing on Windows, take a look below. The following screenshots were taken right after a pull and compile. No files were changed that could alter the behaviour.

The image above shows that for the most part, everything is A-OKAY! We have 1 todo left and this bunch of tests are good. It should be noted that there should actually be one fail. This failure is consistent with macs and linux computers when they run the same set of tests. So there seems to be a missing error… but that’s not all. Right now we’re in a specific test, however, if I were to go up one level …

As you can see, test_FullScreenHarness actually DOES have a fail. To clarify, the order in which this occurred, I actually started out in the folder view and then went into run test_FullScreenHarness to see what was actually failing. The important thing to note is by running test_FullScreenHarness, we actually LOST a test. It didn’t all of a sudden succeed, it just skipped it. Another reason for concern is that the number of tests total up to 76. The actual number of tests that are supposed to be there is 80. The number of tests for mozPointerLock is correct, which means in the tests are missing in test_FullScreenHarness.

Next up

We’re very close to finishing up the tests, the last one is to fix file_cursorPosEvents.html. The purpose is to make sure that once in pointer lock, mouseover, mouseenter, mouseout, mouseleave events are no longer triggered. After a lot of iterations of the file and consulting with Steven, we’re still somewhat stumped as to how to do this properly. At the moment, if you were to run this test, the set of tests will occur twice. This is because the mousemove that is directed to the child is redirected to the parent, which is why the tests happen twice. A solution was proposed to directly send to the child mouseover, mouseenter, mouseout, and mouseleave events to the child. In this case because those events are essentially eliminated, there’s nothing to signify to us that it’s “okay to check if the events occurred now.” The idea to circumvent that was to use setInterval or setTimeout… which probably won’t fly which means we’re still at the drawing board.

Mouse Lock Tests Status

It’s funny, if you were ask me 5 years ago, if I would associate legacy with something good or something bad, I would’ve definitely said good. Legacy… it SOUNDS good, prior to entering the world of programming much of my exposure to the word legacy were in the form of “so-and-so hero left a legacy” again, a good thing. So I was rudely introduced to the fact that often times in the programming world legacy can be equated to outdated and not-maintained.

For all the great teaching and experience I get while studying at Seneca, most of the time, code is created from scratch. Due to this, our projects are almost always small scale assignments, even group projects are relatively small (until professional options anyway). The amount of exposure students get with regards to large code bases, or dealing with existing-but-buggy code is almost nil. Even for projects where the professor suggests sources for code we can use, much of the code is either in copy-and-paste condition or better code can be obtained through a web search where we can THEN copy and paste. Rarely do we have a chance to touch code where there’s not necessarily a “better” alternative and where we just have to buckle down and decode the program in front of us. So reworking some of these mouse lock tests became just that, of course much of it was my fault to begin with…

I’ve done group work in programming classes and I’m tutoring at Seneca’s Learning Centre, so I’ve seen my fair share of coding differences. However, when I started to look over the tests again, wow. There’s a lot to be learned from reading other people’s code, and learn I did. Unfortunately the flip side was that there were a few tests that either tried to do the impossible or tried to do something but was unsuccessful. The problem was the amount of time it took to decode some of the tests that were written in order to determine if the tests needed to be changed, updated, or even removed. Luckily Steven was there to help me and took a little over half of the tests that were peer reviewed to look over.

In the end, I dealt with file_MouseEvents, file_defaultUnlock, test_mousePos, file_movementXY, file_targetOutOfFocus, test_MozPointerLock, file_limitlessScroll, and file_userPref.

There’s no point talking about what changes happened to each individual file since most of the files were actually fairly simple fixes, they just needed minor updates to make sure that the tests worked with the new patch. However, because some of the original authors of the code are no longer working on this project, it’s important for us to understand what it does so when review comes (and we’ve been told to expect the worst), we can get it fixed ASAP.

Looking back the past week at what I said, I underestimated both the amount of free time I would have, as well as how long it would take to fix these tests. The one test that gave me the most trouble was honestly my own, haha. file_MouseEvents.

The main issue was that it was hanging for some, but not for others. As well, some people were having random success and failures. It took me three days and two full rewrites to figure out all the things I did wrong.

During some of the edits I had made, I left some old pieces of code which in turn, caused the test to sometimes stall there until mochitests finally closed it for being open too long. That was problem one taken care off. As for the random success and failures, it took a while for to figure out, but in the end it wasn’t anything I could really do about it. When the test is run, the mouse cursor MUST be in the browser, otherwise, it won’t be able to do the proper synthesis, at least not on Windows. When I launch the test with the cursor hovering anywhere over the browser, the test will work as expected. Take it away and run it? A bunch of failed tests. That’s not really something I can control and fix, when running mochitests, ideally you’re not doing anything that can affect the browser doing its thing anyway.

Finally I noticed a processleak. It was because of this that I had to do the two full rewrites, but every time I got the same thing.

As you see above, that’s a lot of leaking. I always thought it was my code, that something I coded caused this. After a bit of sleuthing, I found that by calling synthesizeMouse and passing in an event of type “mousedown” and with a button of “1” (middle mouse button), mochitest would leak like this. I found another test_bug549170.html, I ran it, and got the similar results. It leaked like mine. This leak seems to only affect Windows machines because Steven, using linux, did not have the same leak issues. And really, I never heard anyone who was peer reviewing this test mentioning the leak (many of them used a linux box to do the builds and testing).

In any case, the majority of the tests ARE done. Steven mentioned two tests that need to be taken care of. The first is one that will be merged with movementXY, and it deals with testing to make sure that the mouse can move more than 1 screen’s length and height once in mouse lock (it’s easy enough to test it out in real life though, just look at the demo pages haha). The next is do something similar to what I’ve done in file_MouseEvents. In file_MouseEvents, I have one outer div that holds a child div. Once mouse lock is set on the outer div, the child div should not get many of the mouse events. The case that is being sought is to have multiple children inside the outer/parent div. It should be fairly straightforward to rig up. =)

I really need to get into the habit of blogging more often, or else I get these REALLY long pages of text.

Open source development – Getting back on track

Originally I had planned on making one massive post but after finishing about 75% of this post, I realized it would be a bit too massive for anyone to bother reading, so I’ve split it into two parts. Part 2 coming up later… by later I mean hopefully with all the remaining tests reviewed, duplicate tests expunged, and all possible errors solved. =) This part will basically detail what I went through to get back on track.

No doubt, a few of you will be wondering how sidetracked could I be when it’s been only 2 weeks since I last worked on open source material, and the answer is very! When Christmas Eve came around I essentially shut off my programming side of my brain and since then I’ve been in the boot up phase. Any one working on open source knows how much things can change in two weeks, heck even a lot can change in one day. There are so many parts that make up Firefox and little adjustments on a part that doesn’t seem related to mouselock can totally break everything regarding it. Of course, I’m working more on the tests side so the impact is softened. Anyway, I’ll start off with when I updated to patch v2 of mouselock.

Building Firefox on Windows last semester was awkward because it forced me to build FF on one core due to random deadlocks when trying to split the workload into multiple jobs. With patch v2 which contained newer mozilla-central code, the way Windows machines build FF has been changed. Now instead of using the normal make, windows users should use pymake and the improvements for switching are very promising. As with many new things, I failed this operation quite quickly. The reason was because my existing object directory was created using make and didn’t play nice with pymake, so after clobbering I was able to perform a full rebuild in just under an hour!

For future reference instead of typing make -f build I have to type python -OO build/pymake/ -f build

Also, after version 2 I knew that vendor prefixes were added so the gh-pages needed a quick update. I took this on because it would help me figure out what I needed to replace on the tests we currently have. It was actually a fairly procedure, and it highlights one of the things I’ll miss due to the new web console interface. When working on the fix to the gh-pages, I was able to start typing navigator.m and it a pop up would show up and tell me instantly what exists under navigator that started with m. Also, when I needed to see what movementX and movementY for mouse events changed to, I could just expose a mouse event via the web console and check out the properties of the event object. The new interface which comes with the new Firefox nightly, while beginner friendly, takes away a few things.

As can be seen, there’s a nice tab(?) that shows up and indicates what commands the web console can take. So now we need to use { in order to get javascript to run. Which wouldn’t be so bad, but since I have to close the block (at least I’m pretty sure I do) all the variables and whatnot I stored in it disappear right after. Perhaps there’s another way, and I sure hope there is, but this limits my ability to expose an object and look at it’s properties without having to use a Javascript loop. Another minor issue is every time I put my cursor there, the tab pops up, which cases all the text to shift over. It’s small, but it’s somewhat annoying to me.

/end rant

The new interface came to me by way of version 3 of the mouselock patch which came one or two days after I did the gh-pages too, which also needed to be and has been updated. In terms of impacting tests there was only one thing, instead of a function isLocked is now an attribute. The other changes, while important don’t affect the creation and updating of the tests.

Part 2 and more specific comments regarding the tests to come!

Continuing on the work

After taking Christmas off, I’ve restarted my efforts in finishing up the mochitests that are required for our implementation of mouselock. Of course a lot of things have changed since then  and other complications have also arisen. The biggest change is that the tests need to be fixed to comply with the changes to the mouselock implementation (vendor prefixes among others). Another thing that has popped up were leaks that were caught by mochitest. Most of these are due to tests ending before Full Screen actually cancels. So aside from those fixes, there are still a few tests that need to be rewritten and reviews done on code.

The first step I think will be making sure the previous reviewed tests are updated, afterwards take care of the remaining tests.

Milestone 0.4: Release wha? Just fixing broken things!

For the last milestone, I’ve been focusing a lot on our implementation of Mouse Lock, and thanks to the great work and effort put in by our class and professor, Dave Humphrey, we got our implementation up for review last week Friday. Incidentally we got a review back, which wasn’t quite as bad as I had expected. A lot of things still left to do, but the first round of feedback was definitely less scathing that I was led to believe. In terms of actual implementation I haven’t really contributed as much as I would like, my major one was getting the mouse to disappear upon the lock method being called, other than that most other things have been small potatoes.

I also served as a sync point for the class’ tests. Originally I had thought this task would be fairly simple. I’ve been doing this sorta thing for a while now, keeping track of Jesse, Diogo, and Dave’s changes. Unfortunately it didn’t scale up very well. When nearly the whole class started to get involved in the test creation, handling the pull requests became quite a mess. Especially when during the merges, one or two pull requests would cause an unexpected failure on the code. In such cases, I had to put aside what I was doing to fix the repository as soon as possible in order to prevent other classmates’ repositories from going kaputz. During this time I was also able to push out a test of my own. My test file_MouseEvents.html deals with whether or not our implemention properly handles all the possible mouse events. For example, events that require the presence of a mouse cursor (i.e., mouseover, mouseout) shouldn’t exist when the mouse is locked. Also user generated events (i.e., mousemove, button presses) should only affect the element that has been locked. It doesn’t matter if event happens to occur right over another element, only the one that the mouse is locked to should get the event. My test ended up having something like 42 tests. It’s really only this big because all the tests need to be done twice. Once when mouse lock is active, and once when it’s not. The reason why I test when it’s not active is because I don’t want future changes to break how the mouse works normally. Its initial completion coincided with Friday, the day we would submit our work for review.

Unfortunately, it would seem that most of the tests did not use asynchronous calls correctly or used some sort of timeout method. This causes random failures and is overall, a hallmark of a poorly done test. Of course, my test was part of that group. orz … While fixing my test (I feel like I made it a lot more complex than it needs to be), I’m trying to also tackle a case where 2 tests would fail for mac users. For some reason, mouseout and mouseleave events were not firing under normal circumstances. I’m not really sure what to make of this, and I’ve made a slight change to my code (instead of going to pixel 100 which is the width of my box, it is now 101) and would love to see if this was the reason why it didn’t work before. Fixing the test itself (remove timing issues by utilizing the success callback) proved quite challenging. Aside from maybe a week in Java, my exposure to threads and thread programming has only been in theory. So when it came down to doing it for real, things didn’t turn out quite so smoothly. That said, it finally works without any wonky behaviours.

I’ve taken up another test, file_limitlessScroll.html which has its fair share of problems. I’m actually going to do a full rewrite. There are a lot of errors in there, but there’s one very big logic failure. All the synthesize mouse calls tell it to go to the same place 3 times. It needs to use ScreenX and Y in its calculations to truly be able to emulate movement. The aim is to have this done by Thursday night and have someone peer review this, because in its current form, limitlessScroll is causing mochitest to hang and fail.

Test wise, things are starting to look up, I have peer reviewed 2 tests, both written/rewritten by northWind, and aside from a few lines that were longer than the 80 char limit, it was good. So that’s 2 tests down. =)

Road to Mouse Lock: Reflections

A couple of days ago (Friday to be exact) Dave wrapped up our work on mouse lock and a couple of tests we have done and put it up for review. Of course this doesn’t mean we’re done. Aside from getting our work reviewed (which Dave has told us will undoubtedly fail), there are still implementations specified on the w3 specs that we haven’t taken care of or are piggybacking off Mozilla’s Full Screen API. There are also a few tests that we’ve left out because they either just hang or don’t do what they need to do, so as anyone can see there’s still a lot left to do. Luckily there are several people who seem to still be interested in working on mouse lock despite the course being over, myself being one of them.

What’s left to do:

  • Finish off the rest of the tests. Should mention that some of these tests may be impossible, we’ve been having trouble getting focus switching to work nicely with all full screen, mouse lock, and mochitest.
  • Write Litmus tests for the tests that aren’t possible for mochitests (determining if the mouse disappears upon successful mouse lock)
  • Start uncoupling from full screen API so we don’t rely on full screen API to work for mouse lock to work.
  • Incorporate the changes we get back from the review

For most of my classmates, this marks the end of having to work on mouse lock, so I want to take this time to reflect on what I’ve done and my thoughts on working on mouse lock. Working of Firefox is incredibly difficult. Just finding the correct place to insert code can be a real challenge. I remember when I wanted to build on top of what’s been done and to make the mouse cursor disappear, it took many hours of looking at a call stack and doing minor manipulations just to see if I’m in the right area. Of course it won’t seem that way when you look at the code… it’s easy to look at the code and say, “Yep, it makes sense to put it there.” For me when I was starting, finding the proverbial sweet spot was quite difficult. Especially before I hooked Firefox up to Visual Studio to debug. Even then, it took me a while of looking at the stack call to figure out where to edit. Aside from finding where to put the code, another area that was incredibly confusing was learning when I could get what objects and what interface I had access to. Perhaps I missed a MDN document regarding how we can tell easily, but I’ve always had trouble figuring out whether or not I can convert one object into another or if my function call to get an object returned the actual object I wanted. However, despite the hardships, I always felt compelled to continue working because what I was doing was something no one else has done, no amount of Googling was going to ever yield an answer to my question. In a way, I felt like I was an explorer searching an unexplored world.

After large parts of the implementations were complete, our class was given a more test oriented task. We needed tests to accompany our patch to ensure we didn’t break anything and that future additions didn’t break our implementations. At this point, because I had inadvertently updated the wiki the day before, I was automatically nominated with making sure all the tests were assigned to a classmate to do. A side effect of this was that I was also put in charge with managing the branch where all the tests will be stored and tested prior to our submission for review. At first I thought the task was going to be easy, because prior to this, I’ve been doing a lot of merge work from Dave, Jesse, and Diogo. So I thought this was going to be similar… boy was I wrong. Initially things started off easy as only three or four people were actually doing their tests and they were, for the most part, available on IRC. Managing the pull requests from them were simple and easy. When everyone else in the class began participating a few problems popped up:

  1. Because IRC was only suggested and not required, when a pull request needed a fix, the main way to contact them was to use github’s pull request comment system. It’s fine for what it is, but unlike messengers, people usually don’t idle on github for very long.
  2. Although information being up on the wiki or on a FAQ, not everyone (in fact it was the majority) reads it, so they make their own templates despite one given to them, and they make pull requests to the wrong repository.
  3. Some people don’t update their repository prior to their pull request despite being told to, which meant that they coded their work based on Firefox code that was many revisions behind.
  4. A side effect from the above is that they had merge conflicts with other people because everyone had to modify the in order for the build tool to incorporate the test.
  5. A lot of people didn’t actually RUN their code before submitting the pull request.

Regrading which of the 5 problems I had the biggest issue with, number 5 beats out all the others by a wide margin. I had thought that it would be standard practice to make sure what you do doesn’t break everybody’s work. It was often not the case. Unfortunately, due to time constraints I usually accepted a pull request if github said it was okay to merge in without looking at the code too closely. In hindsight I should’ve given all the code a quick glance to see if it works, but I had my own test I had to do at the time. Imagine my surprise when upon finishing my test, I update my local repository and find that I can no longer build. Thankfully because we were working only on tests, it was only the makefile that needed to be fixed.

As with most things done for the first time, some tests were not written properly or simply did not test what they were supposed to do. A few of the tests were written in vacuum where the test writer had not come on IRC to discuss the details of their test, so they did not do what was expected of them. This lead to a peer review of the tests which also included the need for a harness for tests that needed full screen. By Friday morning, 4:30am to be exact, about half the tests were converted to the proper harness format and tested what they should be testing. The last two weeks were incredibly exhausting. For the two weeks, I slept around 3-4am in the morning and depending on the day I had to wake up between 7-9am to make it to class. I assume this is how it feels to be in crunch time at a large software company just prior to release.

I’ve learned so much working on mouse lock so far, I’m sure I’ll continue to learn from it as mouse lock implementation gets for fleshed out. The experience I got from working on mouse lock is so uniquely different from anything I’ve experienced before, if I were an RPG character, I can safely say I leveled up.


This week is exam week, and there are still some tests that need to be done. Hopefully, I’ll find the time to take a more in-depth look at one or two tests that don’t have a primary reviewer sometime this week. Otherwise, I think I’ll be spending my free time between semesters mapping out various parts of Firefox, particularly parts that might influence mouse lock implementation.

Road to Mouse Lock: Part 8

It’s been almost a week and a ton of things have happened since the last time I posted. The biggest change is that our tests were more or less assigned to students, which was fine because most if not all the implementation has been done. Incidentally, I was asked to be in charge of consolidating the mochitests. The tests are stored in my github branch mouselock-tests. I was pleasantly surprised by the number of people who managed to complete their tests by Saturday night.

That said, I’ve been only doing quick looks at the code and accepting them, going with the honour system until now. I’m aiming to run through all the tests as well as looking at the code to remove the use of setTimeouts and tests that aren’t quite working.Of course, I’ve had my own tests to write, number 17 and 18.

The Problem:

Writing a mochitest that checked for various mouse events like mousedown, mouseup, mouseover, mouseenter, mouseleave, mouseout, and click. For mouseover, mouseenter, mouseleave, and mouseout, we need to make sure that these events are no longer triggered when we are in mouselock. For mouseup, mousedown, and click, we want to make sure that only the element that is locked can receive these events.

What I’ve Tried:

So for the mouse position related events, I created a div within a div. I full screened and locked the mouse to the larger container div and moved the mouse around the contained div to check the values.

For the mouse click events, again I created a div within a div, but this time the inside div is places right in the middle of the div that is mouse locked. The idea is that when I do any mousedown, mouseup, mouseclick event, only the div that we locked our mouse to catches the event. The inner div that is placed in the middle shouldn’t.

As originally mentioned I tried the createEvent javascript method. However, doing this skips over the event system totally which means that it won’t serve our purposes. So a quick search via mxr yielded a method called synthesizeMouse that served our purpose. While working on this Saturday night, Tentacle and I  (reads so wrong…), found that trying to use synthesizeMouse with the click mouse event would cause an error. The error being that the DOMWindow’s sendmousevent did not support click. The work-around I found was to call sendMouseEvent to handle the click. After that was basically figuring out the workflow of the test (where to add what event listener ) to get the test running.

The Solution:

It works! Compared to many of the tests so far that’s just 1 or 2 tests per file, mine is something like 24 tests. It seems so deceptively simple when you look at the original description but every mousedown and mouseup require a check for left, middle, and right mouse clicks, and having to keep track of the numbers can get really messy.

What’s Next:

The final stretch is here now. The first implementation (where mouselock requires full screen) is almost complete, just one or two dangling things. I plan to look at how mozilla’s full screen api handles focus lost and seeing how we can implement that for mouse lock. As well, there are still some tests remaining that need to be done and I might help do some of those as well.

Road to Mouse Lock: Part 7

So it’s been a while since my last post, work hasn’t stopped, just slowed. My other courses have large assignments that are due coming up so I’ve been focusing getting those done ahead of doing things to implement mouse lock. During the hour of spare time in between my study for a BTP500 test, I whipped up a really quick demo of how mouse lock would work on a canvas. Unfortunately there’s a stretching issue that I haven’t resolved yet, but the basic idea is there. Luckily not long after, during the recent weekend, Jesse made an incredible demo (he added in mouse lock to an existing fps browser demo). I ran it with a debug build, but I don’t think that’s the cause of my problems, but clipping seems to not work as it should at times and the mouse tracking seems to occur even when mouse lock is off, but those are minor details considering the first actual prototype with mouse lock has been implemented. Aside from that I have 2 other things I’m going to talk about…View post

The Problem:

  1. When working on a dual screen monitor, I get odd behaviours from both mouse lock and full screen. When I’m in full screen and mouse locked mode, I’m able to (if I’m fast enough) take focus away from the browser by clicking (usually the right mouse button) on my second monitor and get results that look something like the following:

    Currently, my mouse is on my second monitor, unfortunately the results I get vary a lot when I try to replicate this behaviour. Sometimes everything will work exactly as it should, other times mouse lock and full screen stays turned on, other times it’s just full screen that sticks. I suppose part of the problem with mouse lock staying on despite focus being taken off is partly due to us piggy-backing a lot of our mouse lock exit strategies on full screen. I haven’t really debugged the issue to find out what’s been happening, so the focus of this post is with problem 2 below.
  2. The other problem I’ve been working on is hiding certain mouse events like mouseover, mouseexit, and other non-user generated events. Eventually I’m assuming that suppressing the context menu for the right mouse click will be required as well, but for now the obvious non-user generated mouse events are my target. The code to do this is actually just a simple if check to see if mouse lock is on. The problem is writing the tests for it.

What I’ve tried:

I was able to do a simple in-person test to see if the events were properly hidden, and for the most part, I was successful (I need to track down other non-user generated mouse events).The problem is when I needed to write code that would used for the mochitest framework. Originally I had thought of using the JavaScript createEvent method on the document. However, when I actually tested this out all the events went through even though they should’ve been blocked. At first I thought it was because I didn’t insert my if checks at the right level and that the JS events were somehow being fired further in. So I spent 3 hours looking through nsEventStateManager.cpp looking for where I was going wrong. It seems that wasn’t actually the problem. Had I looked at the JavaScript implementation more closely, I would’ve noticed that the event is attached to the element and the element dispatches the event by itself. This to me, seems like it circumvents the whole event chain that is what normally occurs when you interact with the mouse (and probably why when I created a context menu event, a context menu didn’t pop up).

The Solution:

It would seem I’ll have to look elsewhere for my solution to this problem. I haven’t fully explored what other events I can create, perhaps there is one that exists that would allow me to do this, otherwise, maybe a content script will be necessary (I assume it will work since content scripts are powerful enough for me to check the current cursor’s image). This will going to require a bit more tinkering…

Road to Mouse Lock: Part 6

So after today’s class, the work I’ve been doing has been pulled into the main repository, which means that my work will be tested by all my peers! Hopefully it all works well, if not, I’m here to fix those problems. I’m not totally satisfied with my code just yet, at the moment I do a simple check to see if the browser is in full screen mode. The specs and how it should work in real life is that I would check to make sure the element that I’m locking to is actually the element that’s in full screen. I’m not the only one trying to get at this element, in fact my friend, James Boelen, tried this and has little success. Also, I’ve decided to structure my posts a bit differently (copying Diogo’s style) because I think it’ll help my posts be more to the point and less all-over-the-place.

The Problem:

We want to check to make sure that the element that is in full screen is actually the element that called mouse lock. We have ready for access, the element that calls the lock method of mouse lock and a reference to a nsIDOMWindow object that initiated mouse lock object. Having done much of the code to actually get the mouse cursor to disappear, we also have the following at our disposal,

  • nsIDOMWindow
  • nsPIDOMWindow
  • nsPresContext
  • nsPresShell
  • nsIWidget

After a bit of searching, I found a way to call GetFullScreenElement() what I wanted from the nsPresContext by doing:

nsCOMPtr<nsIDocument> nDoc = presContext->Document();
nsIDocument::Element* check = nDoc->GetFullScreenElement();

As you can see, while I am able to call the method, the result is an nsIDocument::Element. I took this into debug mode in VS and to my relief, if I poke around inside, I see that the element from the method GetFullScreenElement() matches the one passed in from the argument. The problem was how to compare the two, especially since one of them is a nsIDOMElement while the other is a nsIDocument::Element.

What I’ve tried:

I can honestly say that I know very little of the Mozilla code. I’m not sure when it’s appropriate to use do_QueryInterface, or do_GetInterface, or if I should do some sort of casting. So in hopes of hammering this nsIDocument::Element into something more useful, I’ve tried all permutations of do_QueryInterface, do_GetInterface,  and static_cast<>. The types I tried to get it into were Element *, nsGenericHTMLElement, nsIDOMHTMLElement, etc. I can’t really say any more because I must’ve been at this for hours and I would only get errors upon errors. It was interesting because I tried to store GetFullScreenElement in a simple Element *, but the compiler would tell me to use a nsIDocument::Element *. It drove me INSANE because I would open up the object and see nsGenericHTMLElement. It taunts me! It helps having a buddy, James, who suffers the same problem. =)


Don’t have one yet. I believe on Thursday’s class we’ll be talking about this subject and hopefully cracking this case wide open. It’s just as likely that I’m going about this the wrong way, in which case, whoops! Live and learn.

UPDATE: It seems diogo was able to find the answer! Clearly I didn’t dig far enough, didn’t think about using mozilla::dom::Element … rather I saw that once during one of my compile attempts, but didn’t click for me. But that’s one less thing to do for mouse lock!

What’s next:

The obvious thing is to write tests to check the cursor’s current image(?). By image I mean what cursor representation is it. Is it a text, resize, arrow, hand, wait, etc. Most of the stuff online deals with getting that information from using, the problem is we aren’t setting the element’s style. So that route is a bust, I took to IRC (more of our classmates need to use this) and asked Dave for some guidance. He pointed me to content scripts and said that it should have everything I need to do this. I’m new to content scripts, so I’ll be learning it for the first time, hopefully it won’t be too hard.

Another area that needs to be done is, according to the spec under the lock method, restricting the mouse events that are allowed to dispatch. We need to allow only user generated events such as: mousemove, mousedown, mouseup, click, wheel, etc. while suppressing the others. I think we’re at the right level now with nsEventStateManager to do this. I’ll hope to tackle this problem before the week is done, perhaps by Thursday if I finish my other course work early enough.

Finally something more long term is working on a demo. I think I’ll do something like an image viewer. Use processing.js or something to draw a large picture on a canvas, make it full screen and allow the user to “navigate” across the picture. As for the picture, maybe I’ll do a panoramic picture 360º so that they can spin horizontally forever.