Red Hat SummitChapter 2. Creating YAML rules
Each analyzer rule is a set of instructions that are used to analyze source code and detect issues that are problematic for migration.
The analyzer parses user-provided rules, applies them to applications' source code, and generates issues for matched rules. A collection of one or more rules forms a ruleset. Creating rulesets provides a way of organizing multiple rules that achieve a common goal. The analyzer CLI takes rulesets as input arguments.
2.1. YAML rule structure and syntax
MTA rules are written in YAML. Each rule consists of metadata, conditions and actions, which instruct the analyzer to take specified actions when given conditions match.
A YAML rule file in MTA contains one or more YAML rules.
2.1.1. Rule metadata
Rule metadata contains general information about the rule. The structure of metadata is as follows:
- 1
- The ID must be unique within the ruleset to which the rule belongs.
- 2
- See below for a description of the label format.
- 3
effortis an integer value that indicates the level of effort needed to fix this issue.- 4
categorydescribes the severity of the issue for migration. The value can be eithermandatory,optionalorpotential. For a description of these categories, see Rule categories.
2.1.1.1. Rule labels
Labels are key=val pairs specified for rules or rulesets as well as dependencies. For dependencies, a provider adds the labels to the dependencies when retrieving them. Labels on a ruleset are automatically inherited by all the rules that belong to it.
Label format
Labels are specified under the labels field as a list of strings in key=val format as follows:
labels: - "key1=val1" - "key2=val2"
labels:
- "key1=val1"
- "key2=val2"The key of a label can be subdomain-prefixed:
labels: - "konveyor.io/key1=val1"
labels:
- "konveyor.io/key1=val1"The value of a label can be empty:
labels: - "konveyor.io/key="
labels:
- "konveyor.io/key="The value of a label can be omitted. In that case, it is treated as an empty value:
labels: - "konveyor.io/key"
labels:
- "konveyor.io/key"Reserved labels
The analyzer defines some labels that have special meaning as follows:
-
konveyor.io/source: Identifies the source technology to which a rule or a ruleset applies -
konveyor.io/target: Identifies the target technology to which a rule or a ruleset applies
Label selector
The analyzer CLI takes the --label-selector field as an option. It is a string expression that supports logical AND, OR and NOT operations. You can use it to filter-in or filter-out rules by their labels.
Examples:
To filter-in all rules that have a label with the key
konveyor.io/sourceand valueeap6:--label-selector="konveyor.io/source=eap6"To filter-in all rules that have a label with the key
konveyor.io/sourceand any value:--label-selector="konveyor.io/source"To perform logical AND operations on matches of multiple rules using the
&&operator:--label-selector="key1=val1 && key2"To perform logical OR operations on matches of multiple rules using the
||operator:--label-selector="key1=val1 || key2"To perform a NOT operation to filter-out rules that have
key1=val1label set using the!operator:--label-selector="!key1=val1"To group sub-expressions and control precedence using AND:
--label-selector="(key1=val1 || key2=val2) && !val3"
Dependency labels
The analyzer engine adds labels to dependencies. These labels provide additional information about a dependency, such as its programming language and whether the dependency is open-source or internal.
Currently, the analyzer adds the following labels to dependencies:
labels: - konveyor.io/dep-source=internal - konveyor.io/language=java
labels:
- konveyor.io/dep-source=internal
- konveyor.io/language=javaDependency label selector
The analyzer CLI accepts the --dep-label-selector option, which allows filtering-in or filtering-out incidents generated from a dependency by their labels.
For example, the analyzer adds a konveyor.io/dep-source label to dependencies with a value that indicates whether the dependency is a known open-source dependency.
To exclude incidents for all such open-source dependencies, you can use --dep-label-selector as follows:
konveyor-analyzer … --dep-label-selector !konveyor.io/dep-source=open-source
The Java provider in the analyzer can also add an exclude label to a list of packages. To exclude all such packages, you can use --dep-label-selector and the ! operator as follows:
konveyor-analyzer … --dep-label-selector !konveyor.io/exclude
2.1.1.2. Rule categories
mandatory- You must resolve the issue for a successful migration, otherwise, the resulting application is not expected to build or run successfully. An example of such an issue is proprietary APIs that are not supported on the target platform.
optional- If you do not resolve the issue, the application is expected to work, but the results might not be optimal. If you do not make the change at the time of migration, you need to put it on the schedule soon after your migration is completed. An example of such an issue is EJB 2.x code not upgraded to EJB 3.
potential- You need to examine the issue during the migration process, but there is not enough information to determine whether resolving the issue is mandatory for the migration to succeed. An example of such an issue is migrating a third-party proprietary type when there is no directly compatible type on the target platform.
2.1.1.3. Rule Actions
Rules can include 2 types of actions: message and tag. Each rule includes one of them or both.
Message actions
A message action creates an issue with a message when the rule matches. The custom data exported by providers can also be used in the message.
message: "helpful message about the issue"
Example:
- ruleID: test-rule
when:
<CONDITION>
message: Test rule matched. Please resolve this migration issue.
- ruleID: test-rule
when:
<CONDITION>
message: Test rule matched. Please resolve this migration issue.Optionally, a message can include hyperlinks to external URLs that provide relevant information about the issue or a quick fix.
links:
- url: "konveyor.io"
title: "Short title for the link"
links:
- url: "konveyor.io"
title: "Short title for the link"A message can also be a template to include information about the match interpolated through custom variables on the rule.
Tag actions
A tag action instructs the analyzer to generate one or more tags for the application when a match is found. Each string in the tag field can be a comma-separated list of tags. Optionally, you can assign categories to tags.
tag: - "tag1,tag2,tag3" - "Category=tag4,tag5"
tag:
- "tag1,tag2,tag3"
- "Category=tag4,tag5"Example
A tag can be a string or a key=val pair, where the key is treated as a tag category in MTA. Any rule that has a tag action is referred to as a “tagging rule” in this document.
Note that issues are not created for rules that contain only tag actions.
2.1.1.4. Rule conditions
Each rule has a when block, which specifies a condition that needs to be met for MTA to perform a certain action.
The when block contains one condition, but that condition can have multiple conditions nested under it.
when:
<condition>
<nested-condition>
when:
<condition>
<nested-condition>
MTA supports three types of conditions: provider, and, and or.
2.1.1.4.1. Provider conditions
MTA supports multi-language source code analysis. Searching for a specific language in the source code is enabled using the provider condition. This condition defines a search query for a specific language provider. The provider condition also specifies which of the provider’s "capabilities" to use for analyzing the code.
The provider condition has the form <provider_name>.<capability>:
when:
<provider_name>.<capability>
<input_fields>
when:
<provider_name>.<capability>
<input_fields>
The analyzer currently supports the following provider conditions:
-
builtin -
java -
go
2.1.1.4.1.1. builtin provider
builtin is an internal provider that can analyze various files and internal metadata generated by the engine.
This provider has the following capabilities:
-
file -
filecontent -
xml -
json -
hasTags
file
The file capability enables the provider to search for files in the source code that match a given pattern.
when:
builtin.file:
pattern: "<regex_to_match_filenames>"
when:
builtin.file:
pattern: "<regex_to_match_filenames>"filecontent
The filecontent capability enables the provider to search for content that matches a given pattern.
when:
builtin.filecontent:
filePattern: "<regex_to_match_filenames_to_scope_search>"
pattern: "<regex_to_match_content_in_the_matching_files>"
when:
builtin.filecontent:
filePattern: "<regex_to_match_filenames_to_scope_search>"
pattern: "<regex_to_match_content_in_the_matching_files>"xml
The xml capability enables the provider to query XPath expressions on a list of provided XML files. This capability takes 2 input parameters, xpath and filepaths.
json
The json capability enables the provider to query XPath expressions on a list of provided JSON files. Currently, json only takes XPath as input and performs the search on all JSON files in the codebase.
when:
builtin.json:
xpath: "<xpath_expressions>"
when:
builtin.json:
xpath: "<xpath_expressions>" - 1
xpathmust be a valid XPath expression.
hasTags
The hasTags capability enables the provider to query application tags. It queries the internal data structure to check whether the application has the given tags.
when:
# when more than one tags are given, a logical AND is implied
hasTags:
- "tag1"
- "tag2"
when:
# when more than one tags are given, a logical AND is implied
hasTags:
- "tag1"
- "tag2"- 1
- When more than one tags is given, a logical AND is implied.
2.1.1.4.1.2. java provider
The java provider analyzes Java source code.
This provider has the following capabilities:
-
referenced -
dependency.
referenced
The referenced capability enables the provider to find references in the source code. This capability takes two input parameters, pattern and location.
when:
java.referenced:
pattern: "<pattern>"
location: "<location>"
when:
java.referenced:
pattern: "<pattern>"
location: "<location>" The supported locations are the following:
-
CONSTRUCTOR_CALL -
TYPE -
INHERITANCE -
METHOD_CALL -
ANNOTATION -
IMPLEMENTS_TYPE -
ENUM_CONSTANT -
RETURN_TYPE -
IMPORT -
VARIABLE_DECLARATION
dependency
The dependency capability enables the provider to find dependencies for a given application. MTA generates a list of the application’s dependencies, and you can use this capability to query the list and check whether a certain dependency exists for the application within a given range of the dependency’s versions.
when:
java.dependency:
name: "<dependency_name>"
upperbound: "<version_string>"
lowerbound: "<version_string>"
when:
java.dependency:
name: "<dependency_name>"
upperbound: "<version_string>"
lowerbound: "<version_string>" 2.1.1.4.1.3. go provider
The go provider analyzes Go source code. This provider’s capabilities are referenced and dependency.
referenced
The referenced capability enables the provider to find references in the source code.
when: go.referenced: "<regex_to_find_reference>"
when:
go.referenced: "<regex_to_find_reference>"dependency
The dependency capability enables the provider to find dependencies for an application.
when:
go.dependency:
name: "<dependency_name>"
upperbound: "<version_string>"
lowerbound: "<version_string>"
when:
go.dependency:
name: "<dependency_name>"
upperbound: "<version_string>"
lowerbound: "<version_string>" 2.1.1.4.2. Custom variables
Provider conditions can have associated custom variables. You can use custom variables to capture relevant information from the matched line in the source code. The values of these variables are interpolated with data matched in the source code. These values can be used to generate detailed templated messages in a rule’s action (see Message actions). They can be added to a rule in the customVariables field:
2.1.1.5. Logical conditions
The analyzer provides two basic logical conditions, and and or, which enable you to aggregate results of other conditions and create more complex queries.
2.1.1.5.1. and condition
The and condition performs a logical AND operation on the results of an array of conditions. An and condition matches when all of its child conditions match.
when:
and:
- <condition1>
- <condition2>
when:
and:
- <condition1>
- <condition2>Example
Conditions can also be nested within other conditions.
Example
2.1.1.5.2. or condition
The or condition performs a logical OR operation on the results of an array of conditions. An or condition matches when any of its child conditions matches.
when:
or:
- <condition1>
- <condition2>
when:
or:
- <condition1>
- <condition2>Example
2.1.2. Rulesets
A set of rules forms a ruleset. MTA does not require every rule file to belong to a ruleset, but you can use rulesets to group multiple rules that achieve a common goal and to pass the rules to the rules engine.
You can create a ruleset by placing one or more YAML rules in a directory and creating a ruleset.yaml file at the directory root. When you pass this directory as input to the MTA CLI using the --rules option, all rules in this directory are treated as a part of the ruleset defined by ruleset.yaml file.
The ruleset.yaml file stores the metadata of the ruleset.
name: "Name of the ruleset" description: "Description of the ruleset" labels: - key=val
name: "Name of the ruleset"
description: "Description of the ruleset"
labels:
- key=val2.2. Creating a basic YAML rule
This section describes how to create a basic MTA YAML rule. This assumes that you already have MTA installed. See the MTA CLI Guide for installation instructions.
2.2.1. Creating a basic YAML rule template
MTA YAML-based rules have the following basic structure:
when(condition) message(message) tag(tags)
when(condition)
message(message)
tag(tags)Procedure
In the
/home/<USER>/directory, create a file containing the basic syntax for YAML rules as follows:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
2.2.2. Creating a basic YAML ruleset template
If you want to group multiple similar rules, you can create a ruleset for them by placing their files in a directory and creating a ruleset.yaml file at the directory’s root. When you pass this directory as input to the MTA CLI using the --rules option, MTA treats all the files in the directory as belonging to the ruleset defined in the ruleset.yaml file.
Procedure
Create a template for
ruleset.yamlfiles if you want to pass the entire directory using the--rulesoption:name: <RULESET_NAME> description: <RULESET_DESCRIPTION> labels: - key=val
name: <RULESET_NAME>1 description: <RULESET_DESCRIPTION> labels:2 - key=valCopy to Clipboard Copied! Toggle word wrap Toggle overflow
2.2.3. Creating a YAML rule
Each rule file contains one or more YAML rules. Every rule comprises metadata, conditions and actions.
Procedure
Create a
whencondition.The
whencondition of a YAML rule can beprovider,andoror.Create a
providerconditionThe provider condition is used to define a search query for a specific language provider and to invoke a certain capability of the provider.
The condition’s general format is
<provider_name>.<capability>. The condition also has inner fields to specify details of the search. The way you create aprovidercondition and its inner fields depends on which provider you use and which capability you invoke.The table below lists the available providers and their capabilities. Select a provider and its capability that suit the purpose of the rule you want to create. This part of the condition does not contain any of the condition’s fields yet.
Expand Provider Capability Description javareferencedFinds references of a pattern with an optional code location for detailed searches
dependencyChecks whether the application has a given dependency
builtinxmlSearches XML files using XPath queries
jsonSearches JSON files using JSONPath queries
filecontentSearches content in regular files using RegEx patterns
fileFinds files with names matching a given pattern
hasTagsChecks whether a tag is created for the application through a tagging rule
go
referencedFinds references of a pattern
dependencyChecks whether the application has a given dependency
The example below shows a
javaprovider condition that uses thereferencedcapability.Example
when: java.referenced:
when: java.referenced:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Add suitable fields to the
providercondition.The table below lists all available providers, their capabilities, and their fields. Select the fields that belong to the provider and capability that you have chosen. Note that some fields are mandatory.
Expand Provider Capability Field Required? Description javareferencedpatternYes
RegEx pattern
locationNo
Source code location; see below for a list of all supported search locations
dependencynameYes
Name of the dependency
nameregexNo
RegEx pattern to match the name
upperboundNo
Matches version numbers lower than or equal to
lowerboundNo
Matches version numbers greater than or equal to
builtinxmlxpathYes
XPath query
namespacesNo
A map to scope down query to namespaces
filepathsNo
Optional list of files to scope down search
jsonxpathYes
XPath query
filepathsNo
Optional list of files to scope down search
filecontentpatternYes
RegEx pattern to match in content
filePatternNo
Only searches in files with names matching this pattern
filepatternYes
Finds files with names matching this pattern
hasTagsThis is an inline list of string tags. See Tag Action for details on tag format.
goreferencedpatternYes
RegEx pattern
dependencynameYes
Name of the dependency
nameregexNo
RegEx pattern to match the name
upperboundNo
Matches version numbers lower than or equal to
lowerboundNo
Matches version numbers greater than or equal to
The following search locations can be used to scope down
javasearches:- CONSTRUCTOR_CALL
- TYPE
- INHERITANCE
- METHOD_CALL
- ANNOTATION
- IMPLEMENTS_TYPE
- ENUM_CONSTANT
- RETURN_TYPE
- IMPORT
VARIABLE_DECLARATION
The example below shows the
whencondition of a rule that searches for references of a package.Example
when: java.referenced: location: PACKAGE pattern: org.jboss.*when: java.referenced: location: PACKAGE pattern: org.jboss.*Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Create an
ANDorORconditionAn
andcondition matches when all of its child conditions match. Create anandcondition as follows:Copy to Clipboard Copied! Toggle word wrap Toggle overflow An
orcondition matches when any of its child conditions match. Create anorcondition as follows:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
2.2.4. Running an analysis using a custom YAML rule
To run an analysis, use the --rules option in the CLI.
Procedure
To use the rules in a single rule file,
/home/<USER>/rule.yaml, run the following command:mta-cli analyze --input /home/<USER>/data/ --output /home/<USER>/output/ --rules /home/<USER>/rule.yaml
mta-cli analyze --input /home/<USER>/data/ --output /home/<USER>/output/ --rules /home/<USER>/rule.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow where:
-
/home/<USER>/data/- the directory of the source code or binary -
/home/<USER>/output/- the directory for reports (HTML and YAML)
-
-
To use multiple rule files, you need to place them in a directory and to add a
ruleset.yamlfile. Then the directory is treated as a ruleset, and you can pass it as input to the--rulesoption.
Note that if you wish to use the --target or --source option in the CLI, the engine will only select rules that match the label for that target. Therefore, make sure that you have added target or source labels on your rules. See Reserved labels for more details.
2.3. Creating your first YAML rule
This section guides you through the process of creating and testing your first MTA YAML-based rule. This assumes that you have already installed MTA. See Installing and running the CLI in the CLI Guide for installation instructions.
In this example, you will create a rule to discover instances where an application defines a jboss-web.xml file containing a <class-loading> element and to provide a link to the documentation that describes how to migrate the code.
2.3.1. Creating a YAML file for the rule
Create a YAML file for your first rule.
mkdir /home/<USER>/rule.yaml
$ mkdir /home/<USER>/rule.yaml2.3.2. Creating data to test the rule
Create
jboss-web.xmlandpom.xmlfiles in a directory:mkdir /home/<USER>/data/ touch /home/<USER>/data/jboss-web.xml touch /home/<USER>/data/pom.xml
mkdir /home/<USER>/data/ touch /home/<USER>/data/jboss-web.xml touch /home/<USER>/data/pom.xmlCopy to Clipboard Copied! Toggle word wrap Toggle overflow In the
jboss-web.xmlfile you created, paste the following content:Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the
pom.xmlfile you created, paste the following content:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
2.3.3. Creating the rule
MTA YAML-based rules use the following rule pattern:
when(condition) perform(action)
when(condition)
perform(action)Procedure
In the
rule.yamlfile you created, paste the following contents:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Unique ID for your rule, for instance,
jboss5-web-class-loading. - 2
- Text description of the rule.
- 3
- Complete the
whenblock specifying one or more conditions:-
Use the
builtinprovider’s XML capability because this rule checks for a match in an XML file. To match on the
class-loadingelement that is a child ofjboss-web, use the XPath expressionjboss-web/web-loadingas an XML query. In this case, you need just one condition:when: builtin.xml: xpath: jboss-web/class-loadingwhen: builtin.xml: xpath: jboss-web/class-loadingCopy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Use the
- 4
- Helpful message explaining the migration issue. The message is generated in the report when the rule matches. For example:
message: The class-loading element is no longer valid in the jboss-web.xml file.
message: The class-loading element is no longer valid in the jboss-web.xml file.Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 5
- List of string labels for the rule.
- 6
- Number of expected story points to fix this issue.
- 7
- One or more hyperlinks pointing to documentation around the migration issues that you find.
links: - url: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html-single/Migration_Guide/index.html#Create_or_Modify_Files_That_Control_Class_Loading_in_JBoss_Enterprise_Application_Platform_6 title: Create or Modify Files That Control Class Loading in JBoss EAP 6
links: - url: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html-single/Migration_Guide/index.html#Create_or_Modify_Files_That_Control_Class_Loading_in_JBoss_Enterprise_Application_Platform_6 title: Create or Modify Files That Control Class Loading in JBoss EAP 6Copy to Clipboard Copied! Toggle word wrap Toggle overflow The rule is now complete and looks similar to the following:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
2.3.4. Installing the rule
Procedure
Point the CLI to the rule file you created :
–rules /home/<USER>/rules.yaml
–rules /home/<USER>/rules.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow
2.3.5. Testing the rule
Procedure
To test the rule, point the input to the test data you created and pass the rule using the rules option in MTA CLI:
mta-cli analyze --input /home/<USER>/data/ --output /home/<USER>/output/ --rules /home/<USER>/rules.yaml
mta-cli analyze --input /home/<USER>/data/ --output /home/<USER>/output/ --rules /home/<USER>/rules.yaml2.3.6. Reviewing the report
Review the report to be sure that it provides the expected results.
Procedure
Once the analysis is complete, the command outputs the path to the HTML report:
INFO[0066] Static report created. Access it at this URL: URL="file:/home/<USER>/output/static-report/index.html"
INFO[0066] Static report created. Access it at this URL: URL="file:/home/<USER>/output/static-report/index.html"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Open
/home/<USER_NAME>/output/static-report/index.htmlin a web browser.- Navigate to the Issues tab in the left menu.
Verify that the rule is executed:
-
In the Issues table, type
JBoss XMLin the search bar. -
Verify that the issue with the title
Find class loading element in JBoss XML fileis present in the table.
-
In the Issues table, type
- Click the jboss-web.xml link to open the affected file.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}