Hack on MDN: Building useful tool with browser compatibility information

On Friday, March 16, 2018 to Sunday March 18, 2018, 34 people met at Mozilla’s Paris Office to discuss improving MDN’s Browser Compat Data. Amazing results were 221 pull request which improved our data quality and produced, prototyped and improved dashboards and tools.

Hack on MDN Paris 2018, Hackers in action at the Garnier Ballroom.

Hack MDN Events

Hack on MDN was born from the documentation Sprints that the MDN Team organized in 2010 and 2013. This brought together volunteers who worked over the weekend to create and localize MDN content. We expanded the range of sprints in 2014 by inviting other people to join us. Not only did we invite wordsmiths but people who are interested in code and UX design skills.

In 2014 Paris, and 2015 Berlin were the first Hack on MDN events. Although we took some time off, the community spirit and events that they represent were still very much missed. The Hack on MDN Paris in March was one of the two events planned this year. (The next will be in the fall). We decided that March would be a great month to gather an MDN community. The next will take place in the autumn.

Hack on MDN is an unconference that combines hackathon and hackathon. Participants pitch ideas and agree to work on specific tasks, rather than long discussions. These can usually be done in 3 days. The participants choose projects that can be made significant progress in a short time span.

Data on Browser Compatibility

Uniquely, the web platform aims to provide a consistent user experience across different browsers and tools. Your website is created once and works on every device regardless of OS or tool.

This is impossible to do in a connected world that changes constantly. Browsers use different platforms and implement them at different speeds. It’s highly unlikely that all the major players will implement a feature at once, even if they share the same goal. Developers can make educated decisions on which technologies to adopt and which ones to ignore by knowing the support level in their browsers. Developers can make informed decisions about which technologies to use and avoid (e.g., unstable, nonstandard or outdated).

MDN collected the information on browser compatibility over the past decade. We use this data to enhance our reference pages. This information was difficult to integrate into MDN pages and almost impossible to reuse. We decided to convert this data into machine-readable formats a few years back in order for it to be reused.



An example Browser Compatibility Table (BCD), as seen on MDN Web Docs.

Florian Scholz is leading us in migrating the browser compatibility information to a JSON data base. We are currently at about 60% (including all of HTML, HTML, JS and WebExtensions). It is our goal to include all Web APIs, WebDriver and MathML information.

We are also exploring the possibility of reusing compatibility data in other tools. Some may be developed internally, others externally. Our data is published weekly as an NPM package. It will always be up to date, just like MDN uses it. Our first customer is us!

This year’s goal is to collect 100% of MDN compatibility information in JSON and to begin reusing the data in other tools than our inline compatibility tables.

Paris 2018 Event

It was chosen as the theme for March because of its high level of interest and extensive work. Github’s active BCD community provided an opportunity to network with other contributors and made the event a memorable experience.



Demoing was a great way to get into the code.

Thirty-four people from different backgrounds and organisations gathered in the splendid Mozilla Paris office: Mozilla employees (developers, writers, and even managers) from several different teams (MDN, Open Innovation, Web Compat, and WebDriver/Marionette), volunteers, and representatives from Google, Samsung, Microsoft, and the W3C (both on-site and remote).

Florian Scholz, who presented BCD, introduced everyone to the group. People were able to speak freely and get a good idea of what the other people could do. The project pitching was over, and people gathered in groups to begin the work. It was fascinating to see people interact with other individuals who had either pitch an idea or have specific skills. Within a quarter hour, everyone was already deeply involved in hacking.

Each afternoon ended with a demonstration of the completed work. On Saturday morning and Sunday morning, we held lightning talks where everyone could give anything with the aim of stimulating our imaginations.

On Sunday, we concluded with the final demos. The results were truly incredible.

Results

Hack on MDN participants contributed 221 PRs to our repository. There are so many projects that have been completed, it’s impossible to list them all. Here are some highlights.

Visualization tools

Mozilla Tech Speaker, JavaScript hacker Istvan “Flaki” Szmozsanszky has created a tool which displays a compatible table without using the server. It reads the BCD files and creates the table in the browser. It is an essential piece of code which will enable us to embed compatibility tables anywhere, using our Github pull requests. Flaki developed a feature that can edit the JSON on the page, and then generates a PR. She also studied how to visually display the difference between the old and new data.



Flaki’s tool generates a local BCD table as an example.

John Whitlock, from MDN’s Dev Team, and Anthony Maton focused their efforts on building a bot to handle GitHub requests. They also looked at the back-end infrastructure that would allow for easy code maintenance. The team created a new repository , and converted rendering code to plain JS.

Will Bamberg and Eduardo Boucas created a new macro that displayed aggregate data in one table like animation-* CSS properties.

Data migration

We can use the MDN pages, tools and other data that we have more of in JSON format. The more data you have the better. The event saw us migrate just under 60% of the original data and we made substantial progress over the weekend on the 40% remaining.

Jeremie patonnier of Mozilla led Maxime Lo Re, and Sebastian Zartner in migrating most of the SVG elements and their attributes over the weekend. Chris Mills David Ross, Bruno Bruet all did the same for a number of Web APIs. This is an important step in the migration process. The data that was migrated can be roughly equivalent to one quarter of a year’s work. Well done!

Andreas Tolfsen is one of Mozilla’s WebDriver experts. He worked with Chris Mills to add basic WebDriver compatibility information into our repository and to begin documenting WebDriver using MDN.

Data quality

We have data that isn’t perfect. There are data errors, such as features being marked not supported even though they were supported. Missing data can also be caused by our ability to mark unknown data in a different way.

To improve our data quality, several projects were undertaken.

Mark Dittmer of Google created a bridge between the Confluence tool and MDN. His MDN-Confluence tool allows for cross-checking the information from both repositories.

Ada Rose Cannon, Peter O’Shaughnessy both from Samsung developed a tool which produces the initial data set for Samsung Internet. This brings this critical mobile browser into our repository. This dataset is even more fascinating because it can be repurposed to any Chromium browser. We may also be able one day to add QQ and UC information to our repository.

Libby McCormick and Erika Doyle from Microsoft attended remotely from Seattle. They worked together on Edge-related data, including updating EdgeHTML release dates, adding Edge compat data, and other data for WebExtensions.

Scraping tools

Many people were involved in taking data from MDN and other sources, and then using that to create BCD JSON. These tools save time and allow data to be moved faster.

Dominique Hazael Massieux of the W3C created a tool which takes in a WebIDL and creates the skeleton for our BCD. It is very useful in documenting all APIs we create. We only need to change the values later. Dominique has submitted several PRs generated with this tool.

Kayce Basques created the MDN crawler tool. It takes an MDN page, and extracts browser compatibility information from it. Although not all data is possible to be read (manually crafted tables don’t always have the same structure), the tool can extract lots of useful information which can then be modified manually. It is an important time-saver for migration. Kayce published the tool also as a glitch.me service with instructions.

Data reuse with external tools

Eduardo Boucas improved his add-on compat_report which produces a visual compatibleity report within Firefox Dev Tools.


“Screenshot from compat-report byEduardo Boucas” src=”https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/files/2018/03/compat-report-500×409.png”/>

Screen shot of Eduardo Boucas’s compat-report.

Julien Gattelier addressed several issues with his tool Compat-tester and added support for global HTML attributes. A contribute mode was also created that allows users to list features not included in the browser compatibility data. This is a way for potential contributors or users to find missing features.

Dennis Schubert of Mozilla’s Web Compat, together with Julien Gattelier, Kayce Basques and Kayce Basques brainstormed about creating a new tool that would repurpose Julien’s compat tester tool to create a report on web compatibility by crawling important websites.

Others projects

Kadir Topal has created a dashboard that allows us to see the quality of our data, and track any improvements.


“Example output from the data quality dashboard” class=”size-large wp-image-32103″ height=”155″ src=”https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/files/2018/03/Screen-Shot-2018-03-28-at-10.15.01-500×155.png” width=”500″/>

Exemple of data quality dashboard output.

Next?

We have a lot to follow up on: review all PRs, and work with new tools and prototypes into our codebase. This is an excellent problem!

We will keep moving our browser compat information and improving its quality. The better data, the better tools that use it, and MDN web docs, will all be better.

Humanity is the most important outcome from this event. We created new connections and relationships among participants. This will hopefully allow for more awesomeness in the future MDN Web Docs as well as the browser compatibility data project.

Are you interested in getting involved? Are you unsure where to start? To learn more about the MDN Community on Discourse and to see how your contributions can help make MDN even better, visit our Discourse page.

Jean-Yves Perrier

Jean-Yves works as a programme manager for Mozilla’s Developer Outreach team. He was previously an MDN Technical writer specializing in Web platform technologies (HTML and CSS), as well as the MDN Content lead for many years.


Additional articles by JeanYves Perrier…