Vereniging Club management software

Migration from Symfony 2.8 to 4.3

Details

Category: Programming internals

Why not migrate directly?

I developed the first version of the software for a long time and ended up with Symfony version 2.8, after it was actively supported. I tried to migrate to symfony 3.4 using several of the migration guides available, but I failed. I could not get the system up and running again, because there are a lot of dependencies in the versions of the components, mixed with deprecations at a lot of places. To fix this, I started with a completely different approach: Start with a fresh up to date empty Symfony 4.3, and then migrate each file one by one. With this approach, I can also update each file and get the most benefit from new features, e.g. use the service injection / auto wiring also in non-service classes.

Major steps

The following shows all the steps I did to migrate. This might or might not be helpful for others, because it shows the migration for MY software. But I think that these steps can also be useful for others. Roughly I did the following steps:

1. Prepare git and move the existing software to a temporary directory
2. Install a new Symfony 4.3 system
3. Migrate all database entities
4. Replace service class types
5. Migrate fixtures
6. Migrate all tests for the entities
7. Migrate doctrine event subscribers
8. Update the translations
9. Migrate twig filter
10. Migrate parameters
11. Migrate forms
12. Migrate controllers
13. Migrating bootstrap
14. Improve unit tests

Prepare git and move existing software

I did not want to lose the history of each file. Also I need to be able to do changes on the old version during the migration process (as I expect it to take several weeks) and merge those changes to the new version.

git checkout -b symfony4_migration
git push -u origin symfony4_migration
# Now checkout in a different directory
cd ..
mkdir symfony4_migration
cd symfony4_migration
git clone https://gitlab.com/LucHamers/vereniging.git .
# Switch to the new branch
git checkout symfony4_migration
# Copy all files to a temporary directory
mkdir old
# Now move all files to the old directory (this can also be done with an IDE like phpstorm)
for file in $(ls | grep -v 'old'); do git mv $file old; done;
git mv .gitignore old
git add *
git commit
git push

Install fresh Symfony 4 system

Now install a fresh Symfony 4, including some additional packages we will need. We need composer for this, which is installed on my system globally. If you do not have composer, download it from https://getcomposer.org.

# Get the basic symfony system
composer create-project symfony/website-skeleton my-project
# Install some additional components (when the first one asks to execute the recipe, say yes)
composer require stof/doctrine-extensions-bundle
composer require knplabs/doctrine-behaviors
composer require --dev orm-fixtures
# Move all files up one directory, including files starting with a dot (next line)
shopt -s dotglob
mv my-project/* .

Activate the doctrine extensions we need by creating the file config/packages/stof_doctrine_extensions.yaml:

# Read the documentation: https://symfony.com/doc/current/bundles/StofDoctrineExtensionsBundle/index.html
# See the official DoctrineExtensions documentation for more details: https://github.com/Atlantic18/DoctrineExtensions/tree/master/doc/
stof_doctrine_extensions:
    default_locale: en_US
    orm:
        default:
            sluggable: true
            translatable: true
            timestampable: true

Now save this to git with a git add *, git commit and git push

Migrate database entities

Move the entity files to their new location

git mv old/src/Core/DataBundle/Entities/*Repository.php src/Entity
git mv old/src/Core/DataBundle/Entities/*.php src/Entity
git mv old/src/Core/SerialLetterBundle/Entities/*Repository.php src/Entity
git mv old/src/Core/SerialLetterBundle/Entities/*.php src/Entity

Now change all namespaces with their new values:

- namespace Core\DataBundle\Entity;
+ namespace App\Entity;
// or 
+ namespace App\Repository; 

- namespace Core\SerialLetterBundle\Entity; 
+ namespace App\Entity;
// or 
+ namespace App\Repository;

- * @ORM\Entity(repositoryClass="AddressRepository")
+ * @ORM\Entity(repositoryClass="App\Repository\AddressRepository") */

Do that last step in all classes.

Also search for all all occurrences of Core/DataBundle or Core/SerialLetterBundle and replace it with the corresponding values. These are used mostly in calls $entityManager->getRepository. All of these calls have to be replaced like this (don't forget to include the entity classes!):

- $entityManager->getRepository('CoreDataBundle:MemberEntry')->someMethod()
+ $entityManager->getRepository(MemberEntry:class)->someMethod()

When all of this worked, then try if the database can be created.

./bin/console make:migration
./bin/console doctrine:migrations:migrate

For this to work, you have to copy the file .env to .env.local and change the value of DATABASE_URL to your settings.

Replace service class types

Several service classes used at several places have to be replaces by the more general interface of that class. This has to be done in constructors of services, which use autowiring, and for services passed to controler methods. There are also services which have to be replaced by other services. Here is a (not complete) list of the replacements:

Migrate fixtures

 Move the fixtures from old/src/DataBundle/DataFixtures/ORM to src/DataFixtures. Then change the type of the fixture and correct the namespaces:

-namespace Core\DataBundle\DataFixtures\ORM;
+namespace App\DataFixtures;
 
-use Doctrine\Common\DataFixtures\AbstractFixture;
+use App\Entity\Address;
+use Doctrine\Bundle\FixturesBundle\Fixture;
 use Doctrine\Common\DataFixtures\DependentFixtureInterface;
 use Doctrine\Common\Persistence\ObjectManager;
-use Core\DataBundle\Entity\Address;
 
 /**
  * Fixture data for the AddressType class
  */
-class AddressData extends AbstractFixture implements DependentFixtureInterface
+class AddressData extends Fixture implements DependentFixtureInterface
 {
     /**

Migrate entity unit tests

Migrating the tests is mostly the same for all tests. As an example, the following shows all the changes which were needed for the AddressTypeTest. Changes to other tests are simular:

- namespace Core\DataBundle\Tests\Entity;
+ namespace App\Tests\Entity;

- use Core\DataBundle\Entity\AddressType;
- use Core\DataBundle\Services\LogMessageCreator;
- use Doctrine\ORM\EntityManager;
+ use App\Entity\AddressType;
+ use App\Entity\AddressTypeTranslation;
+ use Doctrine\ORM\EntityManagerInterface;
use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;

class AddressTypeTest extends WebTestCase
{
-    /** @var  EntityManager $entityManager */
+    /** @var  EntityManagerInterface $entityManager */
    private $entityManager;

    /**    /**
     */
    public function testIsInUse()
    {
-        $addressType = $this->entityManager->getRepository("CoreDataBundle:AddressType")
+        $addressType = $this->entityManager->getRepository(AddressType::class)
                                           ->findOneBy(['addressType' => "Home address"]);
        $this->assertTrue($addressType->isInUse());

Some entities use event subscribers, e.g. the hash password listener. These subscribers must also be migrated now, otherwise the unit tests for the entities will not word. Beside the chages in namespaces there also is a change in the autowiring of a service class, where the interface should be used instead of the implementation:

 
-namespace Core\DataBundle\Doctrine;
+namespace App\Doctrine;
 
-use Core\DataBundle\Entity\MemberEntry;
+use App\Entity\MemberEntry;
 use Doctrine\Common\EventSubscriber;
 use Doctrine\ORM\Event\LifecycleEventArgs;
-use Symfony\Component\Security\Core\Encoder\UserPasswordEncoder;
+use Symfony\Component\Security\Core\Encoder\UserPasswordEncoderInterface;
 
     /**
      * HashPasswordListener constructor.
-     * @param UserPasswordEncoder $passwordEncoder
+     *
+     * @param UserPasswordEncoderInterface $passwordEncoder
      */
-    public function __construct(UserPasswordEncoder $passwordEncoder)
+    public function __construct(UserPasswordEncoderInterface $passwordEncoder)
     {
         $this->passwordEncoder = $passwordEncoder;
     }

Migrate doctrine event subscribers

Migrating event subsrcibers took me quite some time to figure out why they stopped working in the current environment, especially using the onFlush event. I found out that getting the entity manager in the constructor and getting it from the OnFlushEventArgs returns different objects. This first one worked without problem with symfony 2.8 and doctrine 2.6, but not anymore on symfony 4.3 and doctrine 2.9. So to migrate, remove the constructor and get the entity manager and the unit of work directly from the arguments from the onFlush method:

 class DefaultMembershipTypesListener implements EventSubscriber
 {
-    private $entityManager;
-
-    /**
-     * DefaultMembershipTypesListener constructor.
-     * @param EntityManager $entityManager
-     */
-    public function __construct(EntityManager $entityManager)
-    {
-        $this->entityManager = $entityManager;
-    }
-
-    public function onFlush()
+    public function onFlush(OnFlushEventArgs $eventArgs)
     {
-        $uow = $this->entityManager->getUnitOfWork();
+        $entityManager = $eventArgs->getEntityManager();
+        $uow = $entityManager->getUnitOfWork();

Very important:  when creating new objects in the onFlush method, e.g. calling LogMessageCreator::createLogEntry, then you MUST also use the entity manager from the eventArgs, not the one passed to the constructor of the LogMessageCreator object, when persisting the log message or when fetching related objects which are stored in the log message. Doing this would lead to either not storing the log object at all (because we are adding it to the wrong unit of work) or very strange error messages like "Undefined index: 00000000682852b20000000039d298fa" in file UnitOfWork:2995, when persisting the log message to a different entity manager thant the change type is fetched from.

Update the translations

In symfony 4.3, the transchoice method has been removed from the translator, the transchoice filter has been depricated in twig. This has been replaced by a more powerful, but also more complicated system based on the ICU message format. The documentation on how to use this is a bit confusing and not complete. The following shows how to migrate translations to symfony 4.3:

1. Install the php intl library, which is a wrapper fo rthe icu library.
2. Move the content of all messages translation files for each language to the directory translations into one file per language, calling them messages+intl-icu.en.xlf
3. Change all variables in the content from %some_variable% to {some_variable}
4. If there is a choice, then change it according to the following example:

     <unit id="change_email_body">
       <segment>
         <source>change_email_body</source>
   -     <target>{0}%changedByMember% did the following changes|{1}%changedByMember% did the following changes on %changesOnMember%</target>
   +     <target>{hasChangesOnMember, select, no {{changedByMember} did the following changes} other {{changedByMember} did the following changes on {changesOnMember}}}</target>
       </segment>
     </unit>

   In this example, there are several changes:
   • Change the format of the line to {variable_name, select, condition_value_1 {result string 1} condition value_2 {result string 2} other {default result string}}, where variable_name is the condition variable, select is the type of function we need here (this is like the php switch case). Then the value of case is in front of a {} block, which is returned. Be ware, that there always must be a block other, which is the default of the switch statement. So when you only have 2 cases, then there is only one value and one other block.
   • Copy the translation values, which were previously after the {0} or {1}, into the value/default block
   • Replace the percent symbols of the variables by {}
   When you need pluralisation, this also looks differently then before:

   <unit id="import result first line">
       <segment>
           <source>import result first line</source>
   -           <target>[1]Import file is OK. It contains %numberOfMemberEntries% member entry with|[2,Inf]Import file is OK. It contains %numberOfMemberEntries% member entries with</target>
   +           <target>{count, plural, one {Import file is OK. It contains one member entry with} other {Import file is OK. It contains # member entries with}}</target>
       </segment>
   </unit>

   Here the following changes are needed:
   • Change the format of the line to {variable_name, plural, some number {result string 1} other number {result string 2} other {result string for all other values}}, where variable_name is the condition variable, plural is the type of function we need here. Then the value is in front of a {} block, which is returned. Be ware, that there always must be a block other, which is the default of the switch statement. So when you only have 2 cases, then there is only one value and one other block. The values in front can be either "one", "two" or "=1", "=2", ...
   • Copy the translation values, which were previously after the {0} or {1}, into the value/default block
   • Replace the percent symbols of the variables by {}
   • Be ware: there seems to be a bug in the ICU component (or is it a feature?): it is not possible to use the count variable also as replacement variable. To do this, use # instead of {count}.
   In the twig code, the following changes have to be done:

   - <p>{% transchoice resultNumbers.memberEntries with {'%numberOfMemberEntries%': resultNumbers.memberEntries} %}Import file is OK. It contains %numberOfMemberEntries% member entries with up to{% endtranschoice %}:</p>
   + <p>{% trans with {'count': resultNumbers.memberEntries} %}import result first line{% endtrans %}:</p>
   

5. Replace the transchoice command in the controller or service by the new trans block, where the choice variable has to be added to the variable array:
   

   -            $subject = $this->translator->transChoice('change_email_body', !is_null($this->siteName), ['%changeType%' => $logfileEntry->getChangeType()->getChangeType(), '%siteName%' => $this->siteName[$locale]], 'messages', $locale);
   +            $subject = $this->translator->trans('change_email_body', ['hasChangesOnMember' => is_null($this->changesOnMember) ? 'no' : 'yes', 'changeType' => $logfileEntry->getChangeType()->getChangeType(), 'siteName' => $this->siteName[$locale]], 'messages', $locale);

6. Replace the transchoice filter in the twig template:
   

   -<p>{{ "change_email_body"|transchoice((logFileEntry.ChangesOnMember is not null), {'%changedByMember%': changedByMember, '%changesOnMember%': changesOnMember}, 'messages', locale) }}:</p>
   +<p>{{ "change_email_body"|trans({'hasChangesOnMember': hasChangesOnMember, '%changedByMember%': changedByMember, '%changesOnMember%': changesOnMember}, 'messages', locale) }}:</p>

Migrate twig filters

When you have written your own twig filters, then they too have to be migrated. This is though a very simple process: move the file to its new location, e.g. src/Twig and change the namespace to "App\Twig". Because of the automatic service registration and the base class AbstractExtension, you do not have to register this in the services.yaml file.

Migrate parameters

Symfony 4.3 doesn't use parameters in the file app/config/parameters.yml or app/config/vereniging.yml anymore, but changed to using environment variables. These are stored in the files .env and .env.local, where the first one is the default and the second is a local version, which is not checked in to git, which among others contains the database credentials. You can set the variables via environment variables, which can be used in containers. Here we will only look at the .env file.

As long as the entry in the parameter file contains only a string, integer or bool value, moving it to the .env file is easy. Be ware, that is no longer is possible to use structured parameters like vereniging.base_uri, i.e. using the dot in the parameter name. What is also not possible anymore is to define a parameters as array. If you need this, the value has to be changed to a json string and interpreted where it is needed, e.g. in a controller. Passing an environment variable to a service also needs a somewhat changed syntax. The following table shows all the needed changes:

vereniging.use_academic_titles: true

VERENIGING_USE_ACADEMIC_TITLES=true

vereninging.site_name:
    en: the Vereniging member system

VERENIGING_SITE_NAME={"en":"the Vereniging member system"}

arguments:
    academic_titles: '%vereniging.use_academic_titles%'

arguments:
    academic_titles: '%env(VERENIGING_USE_ACADEMIC_TITLES)%'

There is one special case which was not so easy to migrate. I used the parameter vereniging.mailing_list_update_type in line 2 to be able to select which updater (majordomo or mailman) has to be used in the services.yml:

    app.mailinglist_updater:
        class: Core\DataBundle\Services\%vereniging.mailing_list_update_type%
        arguments: ['@doctrine.orm.entity_manager', '@swiftmailer.mailer.default', '%vereniging.admin_email_address%']

 Doing this with environment files is not possible. To still get the same effect, i.e. to be able to define the type of updater in the environment file, I use a factory to which I pass the type from the environment. How to do this is described here. Now the definition in the services.yaml looks like this (the variable VERENIGING_MAILING_LIST_UPDATER_TYPE defined in the .env file is used here in line 6, the value is passed to the factory to return the correct type):

    app.mailinglist_updater:
        class: App\Service\MailManUpdater
        factory:   ['@App\Service\MailingListUpdaterFactory', createMailingListUpdater]
        arguments:
            $senderAddress: '%env(VERENIGING_ADMIN_EMAIl_ADDRESS)%'
            $type: '%env(VERENIGING_MAILING_LIST_UPDATER_TYPE)%'

Using an environment variable in a controller is a bit different then using a parameter. Instead of $test = $this->getParameter('my_parameter');, you will need to use $test = $_ENV['MY_PARAMETER'];. When the environment variable is an array defines as a json string, you will need to use $test = json_deode($_ENV['MY_PARAMETER'], true);. You can also passt the environment variables as a global variable to twig, as a simple string or even as an array, when the variable is a sjon string in the environment:

twig:
    default_path: '%kernel.project_dir%/templates'
    debug: '%kernel.debug%'
    strict_variables: '%kernel.debug%'
    globals:
        my_variable: '%env(MY_VARIABLE)%'
        # This reads the json string from SITE_NAME and converts it to an associative array
        site_name: '%env(json:VERENIGING_SITE_NAME)%'

Migrate forms

Options

When using a form with choices in Symfony 2.8, there was the option choices_as_values, which switched the keys and values of the passed array. It looked like this:

class MemberListsSelectionType extends AbstractType
{
    /**
     * {@inheritdoc}
    */
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder
            ->add('email', ChoiceType::class, [
                'choices' => [
                    'All' => 'all',
                    'Only with email' => 'with',
                    'Only without email' => 'without',
                    'Only without email in group' => 'without group',
                    ],
                'choices_as_values' => true,
                'empty_data'  => null,
                'expanded' => true,
                'multiple' => false,
                'data' => 'all',
            ])
        ;
    }

The option choices_as_values, has been removed in Symfony 3. But since the default is now to behave as if this option was set, the line can be deleted (see this).

Testing

I had some tests which called the method getName() of the form type. When this method didn't exist, it returned the forms type (method in the base class). The method has been removed from AbstractType, so it also has to be removed from the types and from the tests.

Migrating controllers

Migrating controllers is pretty straightforeward, when you did all the previous stept. In controllers, you need to do at least the following:

• Change the extends Controller to extends AbstractController
• Replace all calls to the container (e.g. $this->get('my_exceptionally_great_service') to use dependency injection. In controllers, this can be done in two ways:
  • Use a constructor to pass the needed services.
  • Pass the services to the action methods, in exactly the same way as in constructors
  
  Whether you pass your services to the constructor or to the action method or even a mix of this, is up to personal taste. I decide this for each controller differently, depending for example on needing the service in private methods (autowiring doesn't work here, so I have to pass the service to the constructor) or having to pass the same services to several actions, where I find it easier to read when passing it to the constructor.
• Replace calls to the method $this->getUser() to passing the user object to the constructor or action method as parameter (?UserInterface $user). See here how this works
• Replace the security checks has_role in the annotations by is_granted (maybe also in twig templates).

Migrating bootstrap

 I have been using Twitter Bootstrap 3. In Symfony 4, the form component can be set to user Bootstrap 4. For this to work, also all other places using Bootstrap 3 have to be updated too. There are very good migration guides to do this. In the following text, I will describe what I had to do the update to Bootstrap 4.

First download the current zip file from https://getbootstrap.com/. From this file, copy the file css/bootstrap.min.css to the directory public/css and js/bootstrap.min.js to public/js. Then update the path in your main template. In vereniging, this is in templates/base.html.twig and in templates/layout.html.twig.

There is an alternative to this: install Bootstrap with composer (composer require twbs/bootstrap) and then use something like Webpack Encore to pack the files you need into the public directory. For my system, I do not use this, I think that is a bit overkill.

I had to make the following changes to get from Bootstrap 3 to 4 in the file templates/layout.html.twig:

- <nav class="navbar navbar-inverse navbar-fixed-top">
+ <nav class="navbar navbar-expand-md navbar-dark fixed-top bg-dark">
- <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar"                 aria-expanded="false" aria-controls="navbar"><span class="sr-only">Toggle navigation</span></button>
+ <button type="button" class="navbar-toggler"          data-toggle="collapse" data-target="#navbarSupportedContent" aria-expanded="false" aria-controls="navbar" aria-label="Toggle navigation"><span class="navbar-toggler-icon"></span></button>
- <div class="navbar-collapse collapse" id="navbar">
+ <div class="collapse navbar-collapse" id="navbar">
- <ul class="nav navbar-nav navbar-right">
+ <div class="navbar-nav ml-auto">
- <a class="navbar-brand" href="/{{ path(url) }}"><span class="glyphicon glyphicon-arrow-right" aria-hidden="true"></span> {{ subpage | trans }}</a>
+ <a class="breadcrumb-item nav-link active" href="/{{ path(url) }}"> {{ subpage | trans }}</a>

In Bootstrap 4, glyphicon cannot be used anymore. I changed to font-awesome to get all the symbols I needed. In the main css file, I added two sections needed for some of the elements shown above:

.footer {
  position: fixed;
  bottom: 0;
  width: 100%;
  /* Set the fixed height of the footer here */
  height: 60px;
  /* Vertically center the text there */
  line-height: 30px; 
  background-color: #FFFFFF;
}

body {
  /* This is needed because Bootstrap 4 changes the default font size to 16 */
  font-size: 14px;
  margin-bottom: 60px;
}

/* In the center of the navigation header show the breadcrump with FontAwesome symbols */
.breadcrumb-item + .breadcrumb-item::before {
  font-family: "FontAwesome";
  /* This is a greater than symbol (>) */
  content: "\f054";
  font-weight: 900;
}

I used Bootstraps panel on all pages. Unfortunately, this has been removed in version 4. In this version, this is called card and the syntax is slightly different. To get about the same look as before, I did the following changes everywhere where panels were used:

- <div class="panel panel-default">
-     <div class="panel-heading">
-         <h3 class="panel-title">Some panel title</h3>
-     </div>
-     <div class="panel-body">
-         <p>Some content</p>
-     </div><!-- panel-body -->
- </div><!-- panel panel-default -->

+ <div class="card mt-2 mb-4">
+     <h6 class="card-header">Some panel title</h6>
+     <div class="card-body">
+         <p>Some content</p>
+     </div><!-- card-body -->
+ </div><!-- card -->

with mt-x and mb-x a spacing to the next elements (mt is the spacing above the card, mb below) can be defined.

After all of this, we can do what wanted to do before all of this: use Bootstrap 4 in Symfony form. To do this, add the following to the file config/packages/twig.yaml:

twig:
    form_themes: ['bootstrap_4_layout.html.twig']

Improving unit tests

When using the WebTestCase, searching for text elements is now a bit easier. We can use the call to $this->assertSelectorTextContains. Change the code like this:

- $this->assertEquals(1, $crawler->filter('h3:contains("Filters")')->count());
+ $this->assertSelectorTextContains('h3', 'Filters');

• Prev
• Next