Fixing WordPress’ Biggest Problem - Supply Chain Security | Karim Marucchi, Joost de Valk

Hi, this is your Silhartia and we are here at OpenSource Summit in Denver, Colorado and today we have two guests Karim Marooi CEO at crowd favorite and Yost the Faulk partner at Emilia Capital. Yos Karim, it's great to have you on the show. Thank you. Thank you. It's my pleasure to host you both folks here. Um I would before we talk about the main topic today which is of course fair package manager project before that I would love to know a bit about you know the company crowd favorite and then of course I will talk to you to learn about the immediate capital so let's start with the with the company absolutely so crowd favorite was originally founded to uh install and customize WordPress for the enterprise so over the years we've done everything from uh tech techrunch to um Disney, W uh AT&T, different companies like that. And today we are one of the uh premier customizers of the of the software within the Fortune 500. Excellent. And uh let's talk about Ailia Capital. So I am the before I started Amelia Capital, I founded a company called Yoast, which build a Yoast SEO plug-in for WordPress. uh we sold that in 2021, made some money and started investing into the WordPress space. Uh so been doing that for the last four years. Um mostly because actually in the WordPress space there's relatively little investment into small companies growing up new stuff. So that's what I started doing together with my wife and Yost is one of the most SEO you know if you look at the project you know for the SEO you by default. So is the name came from in your name as yeah is from my name I if I have to give one piece of advice today it would be not to name your company after yourself because if you end up selling your company someone else ends up owning your name. But yeah no Yosio is a quite successful product. uh we sold it to Newfold Digital in 2021. Uh at that time we had about 12 million websites using it and I think right now it's at 13 14 million websites running that software. Yeah, it's a quite successful open-source project. No, I I do know actually it add one more kind of you know exciting element to this discussion today. Uh which actually creates a very good good segue to the fair package manager looking at your roots when you actually introduce yourself. I had something in the back of but I didn't want to ask like that may be unrelated alto together. Uh so I would like to start that uh the fair package manager project talk about the origin of the project where and why it started. WordPress has historically and it's WordPress is a 22-year-old open source project right and it has historically always worked in a way where every WordPress site connects to WordPress.org to get its updates for WordPress itself, its plugins, its themes, its translations, and a whole lot more. Um, over the course of the well, the end of last year, there was a lot of kurfuffle in our industry when there were a couple of things that happened. At one point, WordPress.org decided to block one of the biggest hosts in the WordPress ecosystem from accessing WordPress.org, which prevented all of their customers from updating their plugins, themes, and WordPress itself. And then a couple days, weeks later, um, one of the plugins on WordPress.org, so an independent open source project was taken over and supplanted by a fork of that project by the WordPress.org uh owner as well. And at that point we as an as an ecosystem all began looking like hey we know that we we weren't always happy with how governance and the WordPress ecosystem worked and how the work why their WordPress ecosystem was run. Um but we really started going like hey we need other solutions. Um, I was looking at it from an investment perspective like this is an entirely unsafe ecosystem to invest in and Kareem had had similar problems with with my day-to-day clients. We had major Fortune50 clients saying, "How can we trust a open- source project that can just decide to replace code? What happens if they didn't decide to actually announce it publicly? It's a supply chain security nightmare and this must be fixed." And we were not alone. There were quite a few people that uh were talking about this and were different groups that were forming like, hey, we need to do something else. And um we both wrote a a blog post at the end of December last year where we said, hey, we need this to change and we have an idea of how to change this. because we could afford to be public about this. Uh a lot of people started contacting us and we luckily were able to connect all these groups together. Um and with all of us together decide like hey we're going to move this forward in in one direction which is where we started with fair. So fair technically is both a protocol and an implementation of that protocol which replaces WordPress.org. it. All these hosts started setting up mirrors when this happened, but all those mirrors were built by themselves and and were basically still dependent on WordPress.org. And what we did with fair is that all these mirrors become repositories of themselves and they connect to each other. So we have this federated network where if one of the uh nodes is cut off from access to WordPress.org, or we can actually route around that like the internet and and create a more stable version of of our supply chain for everyone. In doing so, we've also actually fixed a lot of other problems that we have in the WordPress ecosystem. So, historically, themes and plugins were only findable in the WordPress admin if they were hosted on WordPress.org. We've built a new aggregator service that actually allows plugins and themes to be hosted outside of WordPress.org, which allows for an entire commercial ecosystem to be created on top of that, which did already exist a bit, but it was very hard to be found by users because you could not be found in the in the WordPress admin. If they searched for a plugin, they would only find the free plugins on WordPress.org, but they couldn't find the commercial plugins. So in all of that, fair is a a a project, a protocol, a a a client, a server, and well, it's a lot of work. So today on stage, we announced, hey, we we're setting up a foundation with the Linux Foundation to get funding for all that work we're doing and to make sure that we can govern this together with the wider ecosystem so that instead of the one entity that now controls WordPress.org, will replace that with actual governance um that well allows us all to decide together where we take this. I remember early days Drupal used to be there, Jumla used to there are ton of and WordPress was infamous for being the most insecure one also because it was less about insecurities but a lot of users were there on but the way Drupal was overly complicated you know to run Drupal you have I mean you need a whole data center to run Drupal so but then you folks as you also rightly mentioned auto updates which made a because a lot of folks were not updating you know you know they're like five generation behind those plugins and it it went to the same old software supply chain. a patch was there, it was never applied. But I think that we not entirely fixed that problem yet. But what I'm saying is but the open source get the bad rap. People don't think that hey the patch was not applied. The news cycle will say hey you know what the biggest vulnerability found on WordPress. No it was fixed but nobody applied the patch. Can you talk about first of all when you conceptualize the fear project? First of all the fear is in upper case. Is there any specific connotation meaning behind the name as well? Yeah, so fair stands for um federated and independent repositories and the idea was when we first started talking we were seeing the reaction of what was happening with WordPress.org The hosts were putting up these mirrors and each individual host was doing their own technology to do that. And with a open- source project that has 40% of the web today, imagine each different host having different ways to do updates, the problem you were just speaking about would get worse and worse and you wouldn't have the same experience across all the hosts. So the conversation we had and then brought in other groups as well was how do we make it so we can create a common foundation that keeps the open-source project together. One thing is also unique about WordPress is that of course I can host it on a I can host it on digital ocean wherever I want and then WordPress.com you know you can get you know full you know uh what is the involvement of automatic is it automatic right with the project or not yet they are involved they they're not involved yet we'd love them to be um we've invited them we've we've invited them but um yeah so far there's been no real response Um, of course it there is a bit of a a struggle there in that there are a lot of entities around WordPress. There there is Automatic. There's the founder of WordPress who's also the CEO of Automatic and the leader of the project who's also the owner of WordPress.org which is his private website. So there is a lot of well stuff around that that makes it harder for the wider ecosystem to to do this properly which is actually one of the problems that we're running into as well because with the cyber resilience act in Europe coming up the fact that WordPress.org is a privately owned website which is not an open- source foundation is actually a problem. So we needed that we needed to fix that. We do fix that in in large part with what we've built with fair and what we're building with fair because not all of it's done yet. Um the protocol is pretty much okay but not all of the code has been written to do all the federation. Um, but we needed to tackle a lot of these problems where you could in WordPress right now you can't mark an update as a security release in in WordPress right now there it's not required to have a security point of contact when you submit a plug-in. These are relatively simple things but they've not been done in the project because well nobody cared enough to do them even though the tickets have been there and people have been trying to I myself have been trying to get those tickets through for years for years and now we're getting to the point where well we're getting close and we need we need to fix all of that and luckily the fair team which is a very large team and it's definitely not just us there's 50 developers in our core group and about 200 100 developers around that that are building stuff right now and luckily we can tackle a lot of those problems and make our ecosystem better. We're very lucky that uh the hosting companies are starting to test our current version, our MVP and the tests are coming out very positive. They're giving us great feedback and we're looking forward to later this summer having a finalized version with the AT protocol that is just amazing. So we use um in in WordPress slugs used to be the identifier for what the plugin was and you can only do that if you're only hosting those plugins in one place because then you can control that that slug is unique. So what we do with the the bit that we've taken from the AT protocol. So we've taken their DIDs and we're actually using those to uh uniquely identify plugins and then also the keys that come with that to sign the software. So we've actually introduced code signing which we never had for plugins and themes in WordPress before. We've been talking about that for a decade but it was never implemented. And we can now do all those things. And this means that if you have a DID for your plugin, the DID points to where your plug-in is hosted. So if your plug-in is hosted at WordPress.org now, that's perfectly fine. And then if your plugin is hosted somewhere else suddenly because someone took over because you thought that was a better idea, you can change it in the DID where where the update is coming from. You can just fix that without basically transparently for the end user. It's a we think it's a very good concept. Um and we it's also why um as I said pre-show to you I think that in the end fair could actually be used for more projects because this is a a thing that in itself is a technical advancement that that we could use for other CM open source CMSs or other systems as well. I will talk about that aspect because that is kind of a typical open source story. Linux kernel was created Linux wanted to solve his own problem but it's powering the world before I go there I also want to talk about you know when you're talking about that's not just a small team first of all it's a new project you know link for fun it will grow depending on how it grows uh I I also want to talk about the involvement automatic you know because early days you know a lot of companies they did not get involved with open source you know but later on I mean now you see a lot of the you know big names on industry are there so eventually we might see automatic also there but if you look at the the e ecosystem users a lot of big players who are running their website they go to WordPress.com because very easy they don't have to deal with they just use as a SAS generally that is a perception there can you also talk about when we talk about WordPress.org or where companies are hosting there. So what is the criteria where big organization they choose to have full control and they they choose to have self-hosted versus other hosted I just want to understand the the the scope of the problem so that can we can also see how we can see the ecosystem grow in future so one of the problems we have is the word WordPress is used in a very many different ways right so to make it simple WordPress.org or isn't hosting. It's just the underlying infrastructure for installing packages, modules, plugins in our parliament, themes, templates, uh translations and a lot of other things around that. There is no hosting. It's just that WordPress.com owned by automatic is hosting of one segment of websites. They also have VIP hosting for the enterprise and they have a few other projects and companies along that line. So there's the hosting side and then there's the open source side. The open source side, WordPress.org is what we're hoping to evolve, right? So it's outside of Automatics purview from the way they run their business. It is a founder Matt Mullenig who has been running WordPress.org. But we feel it's time for an evolution for the size of this ecosystem to be able to make it easier for the uh the edge where we're all competing as opposed to the core where we're supposed to be collaborating for that edge to start being able to uh understand how to interact with this infrastructure. Just one more clarity before you know move forward when I look at it you know WordPress.org or I look at is that is the upstream even WordPress.com is a kind of downstream you know and it's WordPress.com is a relatively so WordPress.com is not small but WordPress.com is a relatively small small I want to make those people confused so I want to make sure that you know downstream yeah so when you look at the the market share of WordPress um WordPress is about 40% of websites on the web in those numbers WordPress.com is counted as one website Definitely uh all the others are WordPress.org installs on web hosts around the world or on AWS or on all these other where people run it themselves. So the vast majority of WordPress sites are self-hosted WordPress sites where self-hosted is can mean many things to many people. Yeah, that's where I going because the difference is that I can I can self-host on ais lenode you know and then akamay lenode also offers managed WordPress hosting now they are the one they will managing everything for you so when you are building this community so the whole ecosystem will have these kind of players also who are doing the manage work so can you give us a kind of what kind of community around fair project will look like well I think that our hope is that all of those hosts that offer for WordPress hosting whether it's completely self-hosted, managed or something else will use fair as an underlying framework. Right now, if you install Fair is there's a plugin that you can already install and try it and you'll see that it's for the end user pretty transparent. Nothing changes because it just works and no user sees that the URL where the zip file is coming from is no longer WordPress.org, but it's a fair uh URL. that for end users doesn't really matter. And end users also don't have to be bothered by all of this. Most of them hardly know that they use WordPress, let alone how they get WordPress. And I think that's a good thing because it's a level of technical well technical complexity that they don't have to understand. I think the the the beauty of WordPress is that it's relatively easy to use and that's something that we should definitely keep which is why we chose this approach where we can change the governance around a large part of the ecosystem without breaking up the ecosystem and and basically ripping it in two yeah you don't have to be an engineer to drive a car simp it's a lot make easy now I want to talk about crowd favorites you know of course you mentioned earlier the involvement uh from your perspective, you also mentioned pain points. Yes. How do you want to see this project evolve over time? So the beautiful thing is this is a tool for hosts and other organizations. For instance, you could be a large university and you could create a public node that's available for anybody to be able to get updates and packages from the university just like regular Linux. Right? In other places, um you might have a large international um enterprise company like a big entertainment company who says, "I have tens of thousands of WordPress sites behind my firewall. So I want to create a private node just to update my own servers." So that way I know the provenence of my own code. And that will dramatically reduce even further the total cost of ownership because today they're doing upgrades and patches with containers and that takes developer hours. Developers don't want to be doing updates. We already have an update system. Let's evolve it so that we can use it in the enterprise and that's the pain point. At some point will the package man also start hosting or that will remain with the WordPress? So, we we do plan to allow for some hosting on uh our main server at some point, but we don't necessarily even need that because in the way it currently already works, we could allow a plug-in to be hosted on GitHub and update straight from GitHub because there's really no reason for us to host those ourselves. All all that we need to do is aggregate where all these things are and and get the metadata so that we can show them to users in search results but we don't need the actual files to we don't need to host those ourselves. But then you won't have any control about the security and because that will still your own you know there there you touch on a on a point that's of course like what you immediately immediately need on top of that is a moderation layer where you can say hey these packages are unsafe all of that we're currently working on and and it's we have it this speced out this is very possible actually in very similar ways to how blue sky and other at protocol things already do some of this as Well, so this it's a different way of moderation because you're you're basically using denialists or uh saying, "Hey, the don't use these. Do do use these. This brand is really who it says it it is. This this plug-in comes from yos.blog and uh and is really claimed as such. And we can verify because we've signed the packages that it's actually that thing." Um, but there's two. There's update security and then there's new install security. Like how do you figure out that uh there's not like 10 different versions of a certain plugin and that only one of them is real. Those are things we have solutions for, but they're fairly technical and honestly I can't even really fully explain them. Um, but it there there is stuff there that that we're working on and that it'll it will take a bit of time and it will take moderation as it does right now on WordPress.org too. And I'd like to answer your question a different way as well. Right now we have one company without checks and balances making all these decisions. So one entity. So imagine pick your five favorite hosting companies that have a good reputation. Imagine that all of them are running nodes and they're trusted. One of them decides to go off in a different direction or do something that's untrusted or go offline. There's four others and this becomes exponential. So you have a check and balance system and the reason why we came to the Linux Foundation is to have open transparent governance with checks and balances so that we could work with Linux Foundation members to actually make sure that this is being governed correctly. Linux Foundation actually already has projects around package management. Yeah. So, we were like, we can invent all of this stuff ourselves, which honestly the WordPress world has been very good at doing the last 20 years. We've been very isolationist doing a lot of stuff ourselves in our ecosystem. And we want to break that open and actually collaborate more with the wider open source ecosystem to fix a lot of these things. When you're talking about moderation earlier, what what does it mean for a user like flagging something that it is not safe? If I'm running a WordPress website, as you already said, a majority of WordPress user, of course, there are big organizations, they use it and then there are a lot of mom and pop shops, you know, they are running a small business, they don't even log into the system, you know, that what is going on there. Even if the package is flagged, they don't know. So, they are still running. So no, but honestly right now the situation that situation is the worst. So we can block a plugin on WordPress.org for having a security plugin and you're if you log into your site and look at your plugins page, you would see absolutely nothing. No information. There's no information that that plugin is no longer getting updates. There's no information that that plugin was closed for security reasons. You don't know. Fair protocol changes. we change that and we actually give that feedback to the to the admin so that at that point you can actually see hey this plug-in was closed for security reasons and and you can actually say hey maybe I have to replace this with something else but right now within WordPress this is a deep and utter failure of WordPress itself in my opinion like we don't show you that we don't we literally don't give you that information can I just take it to the next level from the user's perspective that if a user logs into an admin. I am running an Etsy website where I sell something. I never log into back end. I just go put a blog post that new things are there. So how do they I mean I'm just getting too much into details here. But this is the problem you're trying to solve either way. Yeah. So if they log in and then they find out then you have already lost a lot of battle because they have I agree. I would I would love for us to actually fix that and say, "Hey, you know what? We'll send you an email when a plugin that's if I'm running, I should get an email, you know, I get email from Gmail all the time. Hey, you know, you have you have not enabled the two factor authentication. I get three times and I I agree. And that's so that's your long-term I want to understand." Yeah, please go ahead. But I want to understand this bigger vision also you have and that's why it's so important to work with the hosts. Exactly. Because they own the customer, they own the end user and they want this too. So they want to be able to give better service to the end users and what we're trying to do is come up with this common standard for those hosts to be able to do it easily without all having different infrastructures. What are your immediate goals? The good thing is the Linux foundation has a very good structure for governance. So I don't want to talk about the technical committee that procedure is there. But what is your immediate goals? Hey, this is what we need to do now in one or two months and then this is what we want to achieve one you know when we kind of celebrate our anniversary that we look back and they say this is what we achieved. So what is your vision looks like? So immediately over the next month or so we have an open call to the Linux Foundation member companies to come and join us to help create the check and balance system. Then by the end of summer we're hoping to have a 1.0 0 version that is already been these current versions have already been tested by hosts but the 1.0 0 version we're confident is going to be enough for hosts to actually start distributing this to their end users. And a year from now, we're hoping that we've created enough of an ecosystem, these partnerships with these hosts and these large organizations that we have a truly decentralized federated way to evolve the ecosystem to make more possible business and code and evolution of where WordPress is than it has in the last 10 years. Now, whatever you explain, thank you for that. it is it's all look like doing something because you saw a problem it's more or less like you know just voluntary make it better now CRA is coming you know and you know all the companies you you know you mentioned you are from Europe either way uh it won't be any you know and just act up yeah we'll make it so what impact do you see cra you see that actually creating fair share pack actually very well alliance because then companies will fear for big fiance so can you talk about the relevance of this project from that perspective. Well, it's actually why hosts are already engaging with us now as well. Um, they need this they need this to be fixed because this is a an existential threat to to WordPress itself and and honestly right now I think the the way that it's that WordPress.org is set up right now without fair it isn't an existential threat to itself. And with fair we fix that and we we allow all these hosts to do that. So I I'm very confident that European hosts first of all but more hosts in the wider ecosystem because let's be let's be honest CRA is European now but we will probably have similar laws like that in the rest of the world very soon. Everyone needs this. Everyone needs better better supply chain security. Everyone everyone needs open source to actually be open source and governance in in a way that we we all agree can all agree that it's open source and not one person's project. And I so I I expect that uh a lot of hosts will actually do this because it's good business for them, not because they think it's ethically better or something else. It's just a better business decision. And the way WordPress.org stands today, I couldn't use WordPress for my enterprise clients because I would end up being responsible in the CRA and we know that that type of legislation will eventually also come to the United States and it's going to go worldwide. This is needed for the evolution of open source. Yeah. In the morning I was talking to open SS when I have been tracking CR I mean there are already things in China India I mean in US California they already have some things like that so that will become real they are always uh give analogy from the automo industry you don't buy a car and we will install brakes and airbags later on you know no you know so so with the software that should be those things should be baked in before before it's it's leaves the door not after I actually also think that it's an opportunity for open source in general because all of these things bring back the idea of owning your own data, owning your own software, being able to change what what you can what you do and what and where it goes and not being reliant on one company or individual. And I think that that is a a chance for all of us to actually bring back the the whole open- source ethos and say, "Hey, SAS is a is a nice solution, but it it actually has problems, too." And and they're becoming very apparent in in all of these things. So, I think that there's a golden age of open source ahead of us to to actually well do this all together. And I it's one of the things why I'm very excited for us to join the Linux Foundation so that we can do all of that together way more. Karim Yos, thank you so much for joining me today. It was really important discussion because even our website runs on WordPress. So we know the pain point. It's self-hosted. So you know but the thing is they do take good care of it. So I do know the nuances when I talk to my team they talk about and also when you are techie tech journalist that's why you become tech journalist right because you want to go into the so it's really good to bring this point that people are not aware of yes and I do hope and wish that automatic will eventually join because we have seen it pass because we need to build a very secure uh ecosystem around some of these most widely used technologies uh but I really appreciate your effort to create this uh project and also So to learn about your history as well with the Yos project now I know and um as you mentioned that you your next step is call for other parties to join I would love to sit down with you if a video can help to broadcast this mission message and then when the first release is ready we'd love to talk about that which means that there's a lot to talk about uh but I really appreciate time thank you thank you so