Full departure plan from MS Teams

Hello all,

So, I think we are completely ready to depart from MS Teams, and we probably won’t have to lookback.

There are three key components that have been in my mind.

  1. High Availabililty (HA) of server
  2. Mathjax compatibility everywhere we use
  3. LMS (Moodle) - ERP/Chat (ERPNext) - Website (Drupal) integration.

Let me lay it out one by one.

1.High Availability of our servers
I have already set up replica options for all our applications. For below services, at least two server machines are behind the proxy server.

  • Contents: Drupal (going to be main), WordPress (backup)
  • LMS: Moodle for GIAI LMS and SIAI LMS
  • Frappe/ERPNext with Kanban dashboard, Drive, Chat, and student information
  • MediaWiki
  • Discourse (by docker replica)
  • Big Blue Button (by docker replica - and BBB’s proxy configuration Cluster Proxy Configuration | BigBlueButton)
  • Jitsi Meet: For in-house meetings, will be added to Frappe’s chat app

Was going to use docker everywhere, but we’ve got too many standing dev issues that come in #2 and #3, so I will stick to hard machine installations for most of them.

All systems are at least double installed w/ Nginx load balancer and/or rsync (and lsyncd) is running at the back to create instant replica. For DB, MariaDB is running with Galera cluster and PostgreSQL is with HA cluster. Images are now relying on Min.io, a S3 compatible local CDN option that handles replication and load balancing. I also use Cloudflare’s load balancer (and Algo routing) to redirect all traffic to the backup IP in case main server goes down. In short, as long as either primary or backup is running, we are still connected. :smiley:

2. Mathjax compatiblity everywhere we use
As @davidoneill mentioned earlier, this was our initial motivation that we decide to pay days and months of time to ditch MS Teams. It is fully operational on most of platforms in #1 or soon be supported without much trouble. Open source solutions indeed open the door for us to customize (nearly) everything.

Except Jitsi Meet, we will have full Mathjax support by the end of this year.

3. LMS - ERP - Website integration
This has been a major hurdle together with #1, and I can only tell you that we have a plan that likely takes over a year path.

LMS to ERP integration is mostly done, at least ones in our item list. What’s next is how we can enforce students interactively use both Moodle and ERP’s chat. I suspect major actions will take place on Moodle, but without chat support, students likely will feel less integration to school. The ERP solution’s app feature can fill the gap. But as raised by @catherinemaguire, it may not be wise to rely too much on chat. This is a topic that might require years of experience, or YMMV. I was a bit concerned about grading integration, but both Moodle and ERPNext read 3rd party database’s components by hook. We can choose either one platform for studen’s course fee payments. I prefer ERPNext, but will take the team’s opinion on this. In the end, students have to come to the ERP website for their transcripts, academic records, as well as a host of other school related informations like regulations.

LMS to Drupal integration is straight forward. Going forward, all calendar items for education, be it for SIAI or MDSA (supported by GIAI or not), will be initially added to Moodle, and then pulled to Drupal websites by RSS. There actually are plugins for this. ( Calendar | Drupal.org) Course contents can also be pulled by RSS from Moodle. Essentially Moodle is a well-thought application that has done all heavy lifting already. Except highly custom requests, I don’t think we will strain much on this.

ERP - Website integration requires long way to go. For documents, I prefer to use ERPNext’s wiki docs where we can easily set who can read and write. It is not WYSIWYG editor, but Markdown format does have some benefits. I initially thought of Drupal’s dedicated content types for internal / external docs, but given the nature of our institution, there are certain advantages with Markdown format and dev doc style design. It is the team’s joint call.

Push notifications require a gateway, and I will test local installation for the next a few weeks.

For images, I am going to use following options.

  • Sharepoint to Frappe Drive: Internal solution is way faster and more accessible, and this is somewhat Dropbox style that we can have students use for collaboration
  • Min.io with Nginx based custom CNAME CDN

By hook, the ERP’s kanban dashboard and chat can be configured to Drupal node’s update. So, when a new article is created, the dashboard item can be turned to complete or go to next stage of editing, which can also be noti pushed to the chat window. I can also add OpenAI integration to chat so that writers can customize writing within it’s own chat, and when it is done, load the contents on Drupal’s node editing page by Drupal - OpenAI module. Later, when we are done with our own LLM, let’s replace OpenAI dependency.

Full editing integration probably needs some more intense dev work. At least I am not aware of plug and play solution for Gutenberg.js on any Python Frameworks…Once this is done, as @ethanmcgowan said, writers (and even us) don’t have to go to Drupal website anymore. It’s probably only the design team whenever we have major updates.

Pending items are Drupal headless, MediaWiki to Drupal and Moodle (and Disoourse) for keyword/taxonomy integation, and application pages for SIAI (and GIAI) through the ERP’s LMS solution, but these tasks probably require more dev works. We probably have to hire an experienced dev for each function, unless one of you are ready to spend months without sleep.

There still are minor hiccups here and there, but this is pretty much what has been and will have to be done from dev side. Once we are done migration from WordPress in mid Nov, we probably won’t have to check Google’s page speed again. Most solutions give us near 100 without much optimization.

I think it is safe to say that we are ready to move on and focus on operation and teaching.

Deeply thought road map it is @KeithLee

I suggest we use Frappe Wiki for all docs and books. This will further offload design burden and should help them to focus more on detailed customization on other key ERP functions as well as headless Drupal.

We can keep them in one installation with varying permissions, but I actually prefer to have separate installations. For internal documents, whether for company (GIAI) and school (SIAI), it is better to keep them within the main ERP installation. There can be external docs, then we can assign a subdomain. Books is an example of the external doc that it isn’t anything to do with GIAI’s operation. We assign a subdomain and replace the plugin dependency in Wordpress. The doc heavily uses math equations and Markdown. Drupal’s CKEditor or Gutenberg.js are not really meant for that. Re-creating webpage design for books is another matter.

Regarding books design, I am aware that the Frappe wiki docs is not traditional book type UI, but given the contents of the books that are heavily dependent on math and some Markdown, I want to argue that it is the better fit for us.

When it comes to dependency to chat for education, I think it is much related to what you teach and how you teach. Some people may prefer Discourse like forum solution, which is kinda already supported in Moodle. I remember @davidoneill tried once. As much as we need integration btwn Moodle and Chat, we also need integration btwn Moodle and Discourse, or need to find a way to enforce student behavior in chat similar to forum. But since we cannot see immediate benefits, let’s not spend too much time on this.

I will work on migrating books doc from WP to the new independent document subdomain, if @davidoneill and @catherinemaguire are OK with Frappe Wiki’s design for our books. Markdown is not exactly WYSIWYG, but this is the form that I prefer.

I agree with Books being independently running with Frappe Wiki. After all, this is why we first tried it on my cloud server. Now that full Frappe framework is set for our ERP and core of business, I have no reason to object. Besides, I also prefer typing in Markdown format together with Mathjax support. It is not that I disprefer Gutenberg editor, but I will say Markdown is for one task and Gutenberg is for another.

I will set up the server environment, but I think I just have to run bench migrate and change MS Auth redirection endpoints.

Though I agree with the Frappe Wiki replacing Gutenbetg.js, I am not sure about the external installation and URL structure. After all, what we have now is the book for SIAI students. There are chances that we may extend the scope of contents, but for SIAI books, isnt it more obvious to have them in portal.siai.org/books/book_name?

We might have a lot of books in the end, but even for that, I think Frappe wiki is not designed to host thousand books. It gives the wiki home URL for each head document. I suggest we go with purpose driven URL. Besides, one day, we might have smart enough SIAI students to contribute the lecture notes in that docs.

@davidoneill Agreed.

In fact, I always questioned what we can do with previous year exams. With WordPress, we did not have much choice, but now with native Markdown support, I don’t think we should constrain much. We prbbly gotta do little more than just copy&paste for the LaTeX exams, but converting them to Markdown + Mathjax is a walk in the park.

Given the new possibility, I agree that we keep lecture notes to school portal. I was going to say the same for dissertations, but that’s a totally different agenda. I always wanted to put them under mdsa.giai.org, as it is MDSA who evaluates and controls the dissertations. Like our Korean branch with kr.giai.org that @KeithLee is running, matters related to MDSA should be handled separately. We haven’t been able to split due to limitations with WordPress, but with Drupal and Frappe, I fully agree with @davidoneill that we should re-assign URLs by true purpose and assign proper team for each task.

@davidoneill always comes up with new idea that we have disregarded long time ago.

In addition to that, with notes in the students portal, we can also control documents visibility to public and students. I remember ERPNext supports granular control of permissions, which can be exploited.

We might be able to revisit the books subdomain when we are done re-purposing lecture notes. We have so many books to write.

Thx for the thoughts @ethanmcgowan @catherinemaguire @davidoneill

Once the design is ready, I will then migrate lecture notes to the new student portal. Nginx redirection also should be easy.

What’s a bit concerning is the documents’ main page. We probably have to create a summary page like we have seen at Gibbon Edu, docs.gibbonedu.org, which contains 3 docs link. This is purely design work, so other websites can be a bit delayed, unless any of you to contribute shiny Markdown design. :kissing_heart:

Other than that, I am OK with lecture notes’ new location.

And I also agree with @ethanmcgowan that we should assign all dissertations to MDSA’s subdomain. Like my Korean arm, we are affiliates not subsidiaries. I will discuss details with the design. I expect they just combine two pages to one, so shouldn’t be that difficult. Also will speak to MDSA head.

Have spoken to MDSA head, and they seem like to have been nice to me not raising that. Will do so as we move to new platform. prbbly sometime in mid Nov.

Images are now S3 (Min.io) connected for all our platforms.

Might be too early to call, but given the set up and malfunctioning LaTeX on Frappe chat, student ERP won’t be as busy as we expected earlier. SIAI LMS will be the key site that we know, but Moodle chat vs ERP chat is a too tough game for the ERP. Unless we take some measure to boost Frappe chat, students are highly likely stay with MS Teams. Any feedback you got from students, @KeithLee ?

Students are still on Teams, and whatever the early feedback we get, I think we should push forward both chat platforms, at least for the next a few months. @catherinemaguire

Sorry to put them under the test, but this also means that they have a choice, unlike all future students.

My guess is that they will likely be flocked to Moodle, as long as we keep announcements there. ERPNext + Education Module + LMS + Raven is undoubtedly a wonderful combination, but main teaching will likely occur on the more mature platform where we can offer a variety of teaching tools. Unless we jump on to ERPNext’s programming, the platform likely will remain as for new student intro and school management.

Agree. @davidoneill

For student feedback, @catherinemaguire, I will resume teaching from Mid Nov. So by end of this month, I will have some comments from students.

Though ERPNext provides a lot of tools, I think we are better off with Moodle, at least for now. The system completeness for teachin side is beyond comparable.