Publications | Eneldo Loza Mencía

Other (possibly incomplete) resources are DBLP and Google Scholar.

2023

Russa Biswas, Lucie-Aimée Kaffee, Cochez Michael, Stefania Dumbrava, Theis Jendal, Matteo Lissandrini, Vanessa Lopez, Eneldo Loza Mencía, Heiko Paulheim, Harald Sack, Edlira Vakaj, and Gerard de Melo

Knowledge Graph Embeddings: Open Challenges and Opportunities

Transactions on Graph Data and Knowledge, 2023

#knowledge graphs #embeddings #survey

Bib DOI URL

@article{biswas23TGDK,
  author = {Biswas, Russa and Kaffee, Lucie-Aim{\'{e}}e and Michael, Cochez and Dumbrava, Stefania and Jendal, Theis and Lissandrini, Matteo and Lopez, Vanessa and Loza Menc{\'{i}}a, Eneldo and Paulheim, Heiko and Sack, Harald and Vakaj, Edlira and de Melo, Gerard},
  journal = {Transactions on Graph Data and Knowledge},
  title = {Knowledge Graph Embeddings: Open Challenges and Opportunities},
  year = {2023},
  issn = {2942-7517},
  number = {1},
  pages = {4:1--4:32},
  volume = {1},
  address = {Dagstuhl, Germany},
  doi = {10.4230/tgdk.1.1.4},
  keywords = {Knowledge Graphs, KG embeddings, Link prediction, KG applications},
  publisher = {Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  url = {https://drops.dagstuhl.de/entities/document/10.4230/TGDK.1.1.4},
  urn = {urn:nbn:de:0030-drops-194783},
}

2022

Eneldo Loza Mencía, Moritz Kulessa, Simon Bohlender, and Johannes Fürnkranz

Tree-Based Dynamic Classifier Chains

Machine Learning Journal, Mar 2022

#multi-label #dynamic classifier chains #decision trees

Abs Bib DOI URL

Classifier chains are an effective technique for modeling label dependencies in multi-label classification. However, the method requires a fixed, static order of the labels. While in theory, any order is sufficient, in practice, this order has a substantial impact on the quality of the final prediction. Dynamic classifier chains denote the idea that for each instance to classify, the order in which the labels are predicted is dynamically chosen. The complexity of a naive implementation of such an approach is prohibitive, because it would require to train a sequence of classifiers for every possible permutation of the labels. To tackle this problem efficiently, we propose a new approach based on random decision trees which can dynamically select the label ordering for each prediction. We show empirically that a dynamic selection of the next label improves over the use of a static ordering under an otherwise unchanged random decision tree model. In addition, we also demonstrate an alternative approach based on extreme gradient boosted trees, which allows for a more target-oriented training of dynamic classifier chains. Our results show that this variant outperforms random decision trees and other tree-based multi-label classification methods. More importantly, the dynamic selection strategy allows to considerably speed up training and prediction.
@article{loza22DCC, author = {Loza Menc{\'{i}}a, Eneldo and Kulessa, Moritz and Bohlender, Simon and F{\"{u}}rnkranz, Johannes}, title = {Tree-Based Dynamic Classifier Chains}, journal = {Machine Learning Journal}, year = {2022}, month = mar, url = {http://arxiv.org/abs/2112.06672}, publisher = {Springer International Publishing}, doi = {10.1007/s10994-022-06162-3}, }
Michael Rapp, Moritz Kulessa, Eneldo Loza Mencía, and Johannes Fürnkranz

Correlation-Based Discovery of Disease Patterns for Syndromic Surveillance

Frontiers in Big Data, Jan 2022

#rule-learning #syndromic surveillance #pattern discovery #multi-instance learning

Abs Bib DOI URL

Early outbreak detection is a key aspect in the containment of infectious diseases, as it enables the identification and isolation of infected individuals before the disease can spread to a larger population. Instead of detecting unexpected increases of infections by monitoring confirmed cases, syndromic surveillance aims at the detection of cases with early symptoms, which allows a more timely disclosure of outbreaks. However, the definition of these disease patterns is often challenging, as early symptoms are usually shared among many diseases and a particular disease can have several clinical pictures in the early phase of an infection. As a first step toward the goal to support epidemiologists in the process of defining reliable disease patterns, we present a novel, data-driven approach to discover such patterns in historic data. The key idea is to take into account the correlation between indicators in a health-related data source and the reported number of infections in the respective geographic region. In an preliminary experimental study, we use data from several emergency departments to discover disease patterns for three infectious diseases. Our results show the potential of the proposed approach to find patterns that correlate with the reported infections and to identify indicators that are related to the respective diseases. It also motivates the need for additional measures to overcome practical limitations, such as the requirement to deal with noisy and unbalanced data, and demonstrates the importance of incorporating feedback of domain experts into the learning procedure
@article{rapp22diseasepatterns, author = {Rapp, Michael and Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, title = {Correlation-Based Discovery of Disease Patterns for Syndromic Surveillance}, journal = {Frontiers in Big Data}, volume = {4}, year = {2022}, month = jan, url = {https://arxiv.org/abs/2110.09208}, doi = {10.3389/fdata.2021.784159}, issn = {2624-909X}, }
Eyke Hüllermeier, Marcel Wever, Eneldo Loza Mencía, Johannes Fürnkranz, and Michael Rapp

A Flexible Class of Dependence-sensitive Multi-label Loss Functions

Machine Learning Journal, Jan 2022

European Conference of Machine Learning (ECML) Journal Track

#multi-label #non-additive measures #evaluation

Abs Bib DOI URL

Multi-label classification is the task of assigning a subset of labels to a given query instance. For evaluating such predictions, the set of predicted labels needs to be compared to the ground-truth label set associated with that instance, and various loss functions have been proposed for this purpose. In addition to assessing predictive accuracy, a key concern in this regard is to foster and to analyze a learner’s ability to capture label dependencies. In this paper, we introduce a new class of loss functions for multi-label classification, which overcome disadvantages of commonly used losses such as Hamming and subset 0/1. To this end, we leverage the mathematical framework of non-additive measures and integrals. Roughly speaking, a non-additive measure allows for modeling the importance of correct predictions of label subsets (instead of single labels), and thereby their impact on the overall evaluation, in a flexible way - by giving full importance to single labels and the entire label set, respectively, Hamming and subset 0/1 are rather extreme in this regard. We present concrete instantiations of this class, which comprise Hamming and subset 0/1 as special cases, and which appear to be especially appealing from a modeling perspective. The assessment of multi-label classifiers in terms of these losses is illustrated in an empirical study.
@article{wever22mlclosses, author = {H{\"{u}}llermeier, Eyke and Wever, Marcel and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes and Rapp, Michael}, title = {A Flexible Class of Dependence-sensitive Multi-label Loss Functions}, journal = {Machine Learning Journal}, volume = {111}, number = {2}, pages = {713--737}, year = {2022}, month = jan, note = {European Conference of Machine Learning (ECML) Journal Track}, url = {https://arxiv.org/abs/2011.00792}, doi = {10.1007/s10994-021-06107-2}, }

Antonella Plaia, Simona Buscemi, Johannes Fürnkranz, and Eneldo Loza Mencía

Comparing Boosting and Bagging for Decision Trees of Rankings

Journal of Classification, Jan 2022

#ranking #preference learning #boosting #decision trees

Bib DOI URL

@article{plaia22rankingtrees,
  author = {Plaia, Antonella and Buscemi, Simona and F{\"{u}}rnkranz, Johannes and Loza Menc{\'{i}}a, Eneldo},
  title = {Comparing Boosting and Bagging for Decision Trees of Rankings},
  journal = {Journal of Classification},
  volume = {39},
  number = {1},
  pages = {78--99},
  year = {2022},
  url = {https://doi.org/10.1007/s00357-021-09397-2},
  doi = {10.1007/s00357-021-09397-2},
}

2021

Florian Busch, Moritz Kulessa, Eneldo Loza Mencía, and Hendrik Blockeel

Combining Predictions under Uncertainty: The Case of Random Decision Trees

In Discovery Science , Sep 2021

#random decision trees #combination of predictions #uncertainty

Bib DOI URL

@inproceedings{busch21RDTcombination,
  author = {Busch, Florian and Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and Blockeel, Hendrik},
  editor = {Soares, Carlos and Torgo, Lu{\'{i}}s},
  month = sep,
  title = {Combining Predictions under Uncertainty: The Case of Random Decision Trees},
  booktitle = {Discovery Science},
  series = {Lecture Notes in Computer Science},
  volume = {12986},
  year = {2021},
  pages = {78--93},
  publisher = {Springer},
  url = {https://doi.org/10.1007/978-3-030-88942-5\_7},
  doi = {10.1007/978-3-030-88942-5\_7},
}

Moritz Kulessa, Eneldo Loza Mencía, and Johannes Fürnkranz

Revisiting Non-Specific Syndromic Surveillance

In Advances in Intelligent Data Analysis XIX - 19th International Symposium on Intelligent Data Analysis, IDA 2021, Porto, Portugal, April 26-28, 2021, Proceedings , Apr 2021

#outbreak detection #non-specific syndromic surveillance

Bib DOI URL

@inproceedings{kulessa21revisiting,
  author = {Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  editor = {Henriques Abreu, Pedro and Pereira Rodrigues, Pedro and Fern{\'{a}}ndez, Alberto and Gama, Jo{\~{a}}o},
  month = apr,
  title = {Revisiting Non-Specific Syndromic Surveillance},
  booktitle = {Advances in Intelligent Data Analysis {XIX} - 19th International Symposium on Intelligent Data Analysis, {IDA} 2021, Porto, Portugal, April 26-28, 2021, Proceedings},
  year = {2021},
  pages = {128-140},
  publisher = {Springer International Publishing},
  url = {https://arxiv.org/abs/2101.12246},
  doi = {10.1007/978-3-030-74251-5\_11},
}

Moritz Kulessa, Eneldo Loza Mencía, and Johannes Fürnkranz

Revisiting Non-Specific Syndromic Surveillance

CoRR, Jan 2021

Preprint IDA-21

Bib URL

@article{kulessa21revisitingArxiv,
  author = {Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  month = jan,
  title = {Revisiting Non-Specific Syndromic Surveillance},
  journal = {CoRR},
  volume = {abs/2101.12246},
  year = {2021},
  note = {Preprint IDA-21},
  url = {https://arxiv.org/abs/2101.12246},
  archiveprefix = {arXiv},
  eprint = {2101.12246},
}

Moritz Kulessa, Eneldo Loza Mencía, and Johannes Fürnkranz

A Unifying Framework and Comparative Evaluation of Statistical and Machine Learning Approaches to Non-Specific Syndromic Surveillance

Computers - Special Issue Artificial Intelligence for Health, Jan 2021

#outbreak detection #non-specific syndromic surveillance #framework

Abs Bib DOI URL

Monitoring the development of infectious diseases is of great importance for the prevention of major outbreaks. Syndromic surveillance aims at developing algorithms which can detect outbreaks as early as possible by monitoring data sources which allow to capture the occurrences of a certain disease. Recent research mainly concentrates on the surveillance of specific, known diseases, putting the focus on the definition of the disease pattern under surveillance. Until now, only little effort has been devoted to what we call non-specific syndromic surveillance, i.e., the use of all available data for detecting any kind of infectious disease outbreaks. In this work, we give an overview of non-specific syndromic surveillance from the perspective of machine learning and propose a unified framework based on global and local modeling techniques. We also present a set of statistical modeling techniques which have not been used in a local modeling context before and can serve as benchmarks for the more elaborate machine learning approaches. In an experimental comparison of different approaches to non-specific syndromic surveillance we found that these simple statistical techniques already achieve competitive results and sometimes even outperform more elaborate approaches. In particular, applying common syndromic surveillance methods in a non-specific setting seems to be promising.
@article{kulessa21framework, author = {Kulessa, Moritz and Menc{\'{i}}a, Eneldo Loza and F{\"{u}}rnkranz, Johannes}, title = {A Unifying Framework and Comparative Evaluation of Statistical and Machine Learning Approaches to Non-Specific Syndromic Surveillance}, journal = {Computers - Special Issue Artificial Intelligence for Health}, volume = {10}, number = {3}, year = {2021}, issn = {2073-431X}, url = {https://www.mdpi.com/2073-431X/10/3/32}, doi = {10.3390/computers10030032}, article-number = {32}, }
Moritz Kulessa, Bennet Wittelsbach, Eneldo Loza Mencía, and Johannes Fürnkranz

Sum-Product Networks for Early Outbreak Detection of Emerging Diseases

In Artificial Intelligence in Medicine - 19th International Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual Event, June 15-18, 2021, Proceedings , Jan 2021

#outbreak detection #non-specific syndromic surveillance #sum-product networks

Abs Bib DOI URL

Recent research in syndromic surveillance has focused primarily on monitoring specific, known diseases, concentrating on a certain clinical picture under surveillance. Outbreaks of emerging infectious diseases with different symptom patterns are likely to be missed by such a surveillance system. In contrast, monitoring all available data for anomalies allows to detect any kind of outbreaks, including infectious diseases with yet unknown syndromic clinical pictures. In this work, we propose to model the joint probability distribution of syndromic data with sum-product networks (SPN), which are able to capture correlations in the monitored data and even allow to consider environmental factors, such as the current influenza infection rate. Conversely to the conventional use of SPNs, we present a new approach to detect anomalies by evaluating p-values on the learned model. Our experiments on synthetic and real data with synthetic outbreaks show that SPNs are able to improve upon state-of-the-art techniques for detecting outbreaks of emerging diseases.
@inproceedings{kulessa21NSS-SPN, author = {Kulessa, Moritz and Wittelsbach, Bennet and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Tucker, Allan and Henriques Abreu, Pedro and Cardoso, Jaime and Pereira Rodrigues, Pedro and Ria{\~{n}}o, David}, title = {Sum-Product Networks for Early Outbreak Detection of Emerging Diseases}, booktitle = {Artificial Intelligence in Medicine - 19th International Conference on Artificial Intelligence in Medicine, {AIME} 2021, Virtual Event, June 15-18, 2021, Proceedings}, booktitle_short = {Artificial Intelligence in Medicine (AIME)}, series = {Lecture Notes in Computer Science}, volume = {12721}, year = {2021}, pages = {61--71}, publisher = {Springer International Publishing}, isbn = {978-3-030-77211-6}, url = {https://doi.org/10.1007/978-3-030-77211-6\_7}, alt-url = {https://sci-hub.yncjkj.com/https://link.springer.com/chapter/10.1007/978-3-030-77211-6\_7}, doi = {10.1007/978-3-030-77211-6\_7}, }
Michael Rapp, Eneldo Loza Mencía, Johannes Fürnkranz, and Eyke Hüllermeier

Gradient-Based Label Binning in Multi-Label Classification

In Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III , Jan 2021

#multi-label #rule-learning #gradient boosting #efficiency

Abs Bib DOI URL

In multi-label classification, where a single example may be associated with several class labels at the same time, the ability to model dependencies between labels is considered crucial to effectively optimize non-decomposable evaluation measures, such as the Subset 0/1 loss. The gradient boosting framework provides a well-studied foundation for learning models that are specifically tailored to such a loss function and recent research attests the ability to achieve high predictive accuracy in the multi-label setting. The utilization of second-order derivatives, as used by many recent boosting approaches, helps to guide the minimization of non-decomposable losses, due to the information about pairs of labels it incorporates into the optimization process. On the downside, this comes with high computational costs, even if the number of labels is small. In this work, we address the computational bottleneck of such approach – the need to solve a system of linear equations – by integrating a novel approximation technique into the boosting procedure. Based on the derivatives computed during training, we dynamically group the labels into a predefined number of bins to impose an upper bound on the dimensionality of the linear system. Our experiments, using an existing rule-based algorithm, suggest that this may boost the speed of training, without any significant loss in predictive performance.
@inproceedings{rapp21labelbinning, author = {Rapp, Michael and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes and H{\"{u}}llermeier, Eyke}, editor = {Oliver, Nuria and P{\'{e}}rez{-}Cruz, Fernando and Kramer, Stefan and Read, Jesse and Antonio Lozano, Jos{\'{e}}}, keywords = {Gradient boosting, multilabel classification, Rule Learning}, title = {Gradient-Based Label Binning in Multi-Label Classification}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, {ECML} {PKDD} 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part {III}}, booktitle_short = {Machine Learning and Knowledge Discovery in Databases (ECML-PKDD)}, year = {2021}, series = {Lecture Notes in Computer Science}, volume = {12977}, pages = {462--477}, publisher = {Springer}, doi = {10.1007/978-3-030-86523-8\_28}, url = {https://arxiv.org/abs/2106.11690}, }

Michael Rapp, Moritz Kulessa, Eneldo Loza Mencía, and Johannes Fürnkranz

Correlation-based Discovery of Disease Patterns for Syndromic Surveillance

CoRR, Jan 2021

Bib URL

@article{rapp21diseasepatternsarxiv,
  author = {Rapp, Michael and Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  title = {Correlation-based Discovery of Disease Patterns for Syndromic Surveillance},
  journal = {CoRR},
  volume = {abs/2110.09208},
  year = {2021},
  url = {https://arxiv.org/abs/2110.09208},
  eprinttype = {arXiv}
}

2020

Moritz Kulessa, Eneldo Loza Mencía, and Johannes Fürnkranz

Improving the Fusion of Outbreak Detection Methods with Supervised Learning

In Computational Intelligence Methods for Bioinformatics and Biostatistics - 16th International Meeting, CIBB 2019, Bergamo, Italy, September 4-6, 2019, Revised Selected Papers , Bergamo, Italy, Dec 2020

#outbreak detection #stacking

Bib DOI

@inproceedings{mk:CIBB2019,
  author = {Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  keywords = {Fusion, Outbreak Detection, Stacking, Surveillance},
  month = dec,
  title = {Improving the Fusion of Outbreak Detection Methods with Supervised Learning},
  booktitle = {Computational Intelligence Methods for Bioinformatics and Biostatistics - 16th International Meeting, {CIBB} 2019, Bergamo, Italy, September 4-6, 2019, Revised Selected Papers},
  series = {Lecture Notes in Computer Science},
  volume = {12313},
  year = {2020},
  pages = {55--66},
  publisher = {Springer},
  location = {Bergamo, Italy},
  isbn = {978-3-030-63060-7},
  doi = {10.1007/978-3-030-63061-4_6},
}

Andreas Hüfner, Daniel Kiefl, Markus Baacke, Rebecca Zöllner, Eneldo Loza Mencía, Oliver Schellein, Nermin Avan, and Sylvia Pommerl

Risikostratifizierung durch Implementierung und Evaluation eines COVID-19-Scores

Medizinische Klinik - Intensivmedizin und Notfallmedizin, Nov 2020

#COVID-19 #interpretable models

Abs Bib DOI URL

Die COVID-19-Erkrankung (Coronavirus Disease 2019) ist in ihrer klinischen Manifestation unspezifisch und sehr variabel, wodurch die Diskriminationsfähigkeit zu anderen (Virus-)Infekten erschwert ist. Weder Einzelbefunde noch Befundkombinationen sind spezifisch genug, um eine COVID-19-Erkrankung mit hoher Sicherheit diagnostizieren zu können. Ziel war es, in der Notaufnahme durch Anwendung eines Scores frühzeitig Patienten identifizieren zu können, die ein Risiko für eine COVID-19-Erkrankung haben und somit präemptiv isoliert werden müssen.
@article{huefner20covid, author = {H{\"{u}}fner, Andreas and Kiefl, Daniel and Baacke, Markus and Z{\"{o}}llner, Rebecca and Loza Menc{\'{i}}a, Eneldo and Schellein, Oliver and Avan, Nermin and Pommerl, Sylvia}, month = nov, title = {Risikostratifizierung durch Implementierung und Evaluation eines COVID-19-Scores}, journal = {Medizinische Klinik - Intensivmedizin und Notfallmedizin}, volume = {115}, year = {2020}, pages = {132–-138}, issn = {2193-6226}, url = {https://link.springer.com/content/pdf/10.1007/s00063-020-00754-4.pdf}, doi = {10.1007/s00063-020-00754-4}, }
Simon Bohlender, Eneldo Loza Mencía, and Moritz Kulessa

Extreme Gradient Boosted Multi-label Trees for Dynamic Classifier Chains

In Discovery Science - 23rd International Conference, DS 2020, Thessaloniki, Greece, October 19-21, 2020, Proceedings , Oct 2020

#multi-label #dynamic classifier chains #gradient boosting #decision trees

Abs Bib DOI URL

Classifier chains is a key technique in multi-label classification, since it allows to consider label dependencies effectively. However, the classifiers are aligned according to a static order of the labels. In the concept of dynamic classifier chains (DCC) the label ordering is chosen for each prediction dynamically depending on the respective instance at hand. We combine this concept with the boosting of extreme gradient boosted trees (XGBoost), an effective and scalable state-of-the-art technique, and incorporate DCC in a fast multi-label extension of XGBoost which we make publicly available. As only positive labels have to be predicted and these are usually only few, the training costs can be further substantially reduced. Moreover, as experiments on eleven datasets show, the length of the chain allows for a more control over the usage of previous predictions and hence over the measure one want to optimize.
@inproceedings{bohlender20XDCC, author = {Bohlender, Simon and Loza Menc{\'{i}}a, Eneldo and Kulessa, Moritz}, month = oct, title = {Extreme Gradient Boosted Multi-label Trees for Dynamic Classifier Chains}, booktitle = {Discovery Science - 23rd International Conference, {DS} 2020, Thessaloniki, Greece, October 19-21, 2020, Proceedings}, series = {Lecture Notes in Computer Science}, volume = {12323}, year = {2020}, pages = {471--485}, publisher = {Springer International Publishing}, institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt}, url = {https://arxiv.org/abs/2006.08094}, doi = {10.1007/978-3-030-61527-7_31}, }
Vu-Linh Nguyen, Eyke Hüllermeier, Michael Rapp, Eneldo Loza Mencía, and Johannes Fürnkranz

On Aggregation in Ensembles of Multilabel Classifiers

In Discovery Science , Oct 2020

#multi-label #aggregation #ensembles

Abs Bib DOI URL

While a variety of ensemble methods for multilabel classification have been proposed in the literature, the question of how to aggregate the predictions of the individual members of the ensemble has received little attention so far. In this paper, we introduce a formal framework of ensemble multilabel classification, in which we distinguish two principal approaches: "predict then combine" (PTC), where the ensemble members first make loss minimizing predictions which are subsequently combined, and "combine then predict" (CTP), which first aggregates information such as marginal label probabilities from the individual ensemble members, and then derives a prediction from this aggregation. While both approaches generalize voting techniques commonly used for multilabel ensembles, they allow to explicitly take the target performance measure into account. Therefore, concrete instantiations of CTP and PTC can be tailored to concrete loss functions. Experimentally, we show that standard voting techniques are indeed outperformed by suitable instantiations of CTP and PTC, and provide some evidence that CTP performs well for decomposable loss functions, whereas PTC is the better choice for non-decomposable losses.
@inproceedings{ln:MLCaggregation, author = {Nguyen, Vu-Linh and H{\"{u}}llermeier, Eyke and Rapp, Michael and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Appice, Annalisa and Tsoumakas, Grigorios and Manolopoulos, Yannis and Matwin, Stan}, keywords = {Combine then Predict, Ensembles of Multilabel Classifiers, F-measure, Hamming loss, Predict then Combine, Subset 0/1 loss}, month = oct, title = {On Aggregation in Ensembles of Multilabel Classifiers}, booktitle = {Discovery Science}, series = {Lecture Notes in Computer Science}, volume = {12323}, year = {2020}, pages = {533--547}, publisher = {Springer International Publishing}, address = {Cham}, isbn = {978-3-030-61527-7}, url = {https://arxiv.org/abs/2006.11916}, doi = {https://doi.org/10.1007/978-3-030-61527-7_35}, }
Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Michael Rapp

Learning Structured Declarative Rule Sets – A Challenge for Deep Discrete Learning

In 2nd Workshop on Deep Continuous-Discrete Machine Learning (DeCoDeML) , Sep 2020

#rule-learning #multi-label #deep models #position paper

Abs Bib URL

Arguably the key reason for the success of deep neural networks is their ability to autonomously form non-linear combinations of the input features, which can be used in subsequent layers of the network. The analogon to this capability in inductive rule learning is to learn a structured rule base, where the inputs are combined to learn new auxiliary concepts, which can then be used as inputs by subsequent rules. Yet, research on rule learning algorithms that have such capabilities is still in their infancy, which is - we would argue - one of the key impediments to substantial progress in this field. In this position paper, we want to draw attention to this unsolved problem, with a particular focus on previous work in predicate invention and multi-label rule learning
@inproceedings{fuernkranz20decodeml, author = {F{\"{u}}rnkranz, Johannes and H{\"{u}}llermeier, Eyke and Loza Menc{\'{i}}a, Eneldo and Rapp, Michael}, month = sep, title = {Learning Structured Declarative Rule Sets -- A Challenge for Deep Discrete Learning}, booktitle = {2nd Workshop on Deep Continuous-Discrete Machine Learning (DeCoDeML)}, year = {2020}, url = {https://sites.google.com/view/decodeml-workshop-2020/program}, }
Eyke Hüllermeier, Johannes Fürnkranz, and Eneldo Loza Mencía

Conformal Rule-Based Multi-label Classification

In KI 2020: Advances in Artificial Intelligence , Sep 2020

#multi-label #rule-learning #uncertainty

Abs Bib DOI URL

We advocate the use of conformal prediction (CP) to enhance rule-based multi-label classification (MLC). In particular, we highlight the mutual benefit of CP and rule learning: Rules have the ability to provide natural (non-)conformity scores, which are required by CP, while CP suggests a way to calibrate the assessment of candidate rules, thereby supporting better predictions and more elaborate decision making. We illustrate the potential usefulness of calibrated conformity scores in a case study on lazy multi-label rule learning.
@inproceedings{huellermeier20conformal, author = {H{\"{u}}llermeier, Eyke and F{\"{u}}rnkranz, Johannes and Loza Menc{\'{i}}a, Eneldo}, editor = {Schmid, Ute and Kl{\"{u}}gl, Franziska and Wolter, Diedrich}, month = sep, title = {Conformal Rule-Based Multi-label Classification}, booktitle = {KI 2020: Advances in Artificial Intelligence}, series = {Lecture Notes in Computer Science}, volume = {12325}, year = {2020}, publisher = {Springer, Cham}, isbn = {978-3-030-58284-5}, url = {https://arxiv.org/abs/2007.08145}, doi = {10.1007/978-3-030-58285-2_25}, }
Eyke Hüllermeier, Johannes Fürnkranz, Eneldo Loza Mencía, Vu-Linh Nguyen, and Michael Rapp

Rule-Based Multi-label Classification: Challenges and Opportunities

In Rules and Reasoning , Aug 2020

#multi-label #rule-learning

Abs Bib DOI URL

In the context of multi-label classification (MLC), rule-based learning algorithms have a number of appealing properties that are not, at least not as a whole, shared by other approaches. This includes the potential interpretability of rules, their ability to model (local) label dependencies in a flexible way, and the facile customization of a predictor to different loss functions. In this paper, we present a modular framework for rule-based MLC and discuss related challenges and opportunities for multi-label rule learning.
@inproceedings{huellermeier20MLRLchallenges, author = {H{\"{u}}llermeier, Eyke and F{\"{u}}rnkranz, Johannes and Loza Menc{\'{i}}a, Eneldo and Nguyen, Vu-Linh and Rapp, Michael}, editor = {Guti{\'{e}}rrez-Basulto, V{\'{i}}ctor and Kliegr, Tom{\'{a}}s and Soylu, Ahmet and Giese, Martin and Roman, Dumitru}, keywords = {multilabel classification, Rule Learning}, month = aug, title = {Rule-Based Multi-label Classification: Challenges and Opportunities}, booktitle = {Rules and Reasoning}, series = {Lecture Notes in Computer Science}, volume = {12173}, year = {2020}, pages = {3--19}, publisher = {Springer International Publishing}, isbn = {978-3-030-57977-7}, url = {https://link.springer.com/chapter/10.1007/978-3-030-57977-7_1}, doi = {https://doi.org/10.1007/978-3-030-57977-7_1}, }
Simon Bohlender, Eneldo Loza Mencía, and Moritz Kulessa

Extreme Gradient Boosted Multi-label Trees for Dynamic Classifier Chains

Jun 2020

Preprint of https://ke-tud.github.io/bibtex/publications/show/3211

Abs Bib URL

Classifier chains is a key technique in multi-label classification, since it allows to consider label dependencies effectively. However, the classifiers are aligned according to a static order of the labels. In the concept of dynamic classifier chains (DCC) the label ordering is chosen for each prediction dynamically depending on the respective instance at hand. We combine this concept with the boosting of extreme gradient boosted trees (XGBoost), an effective and scalable state-of-the-art technique, and incorporate DCC in a fast multi-label extension of XGBoost which we make publicly available. As only positive labels have to be predicted and these are usually only few, the training costs can be further substantially reduced. Moreover, as experiments on eleven datasets show, the length of the chain allows for a more control over the usage of previous predictions and hence over the measure one want to optimize.
@techreport{bohlender20XDCC_arxiv, author = {Bohlender, Simon and Loza Menc{\'{i}}a, Eneldo and Kulessa, Moritz}, month = jun, title = {Extreme Gradient Boosted Multi-label Trees for Dynamic Classifier Chains}, type = {ArXiv e-prints}, number = {2006.08094 [cs.LG]}, year = {2020}, institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt}, note = {Preprint of https://ke-tud.github.io/bibtex/publications/show/3211}, url = {https://arxiv.org/abs/2006.08094}, archiveprefix = {arXiv}, eprint = {1908.03032}, primaryclass = {cs.LG} }
Margot Mieskes, Eneldo Loza Mencía, and Tim Kronsbein

A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation

In Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC 2020) , May 2020

Data set available at \urlhttps://github.com/keelm/DIP-SumEval

#automatic text summarization #dataset #human evaluation

Abs Bib URL

Automatic evaluation of summarization focuses on developing a metric to represent the quality of the resulting text. However, text qualityis represented in a variety of dimensions ranging from grammaticality to readability and coherence. In our work, we analyze the depen-dencies between a variety of quality dimensions on automatically created multi-document summaries and which dimensions automaticevaluation metrics such as ROUGE, PEAK or JSD are able to capture. Our results indicate that variants of ROUGE are correlated tovarious quality dimensions and that some automatic summarization methods achieve higher quality summaries than others with respectto individual summary quality dimensions. Our results also indicate that differentiating between quality dimensions facilitates inspectionand fine-grained comparison of summarization methods and its characteristics. We make the data from our two summarization qualityevaluation experiments publicly available in order to facilitate the future development of specialized automatic evaluation methods.
@inproceedings{mm:DIP-SumEval, author = {Mieskes, Margot and Loza Menc{\'{i}}a, Eneldo and Kronsbein, Tim}, month = may, title = {A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation}, booktitle = {Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC 2020)}, year = {2020}, pages = {6690–-6699}, publisher = {European Language Resources Association}, address = {Marseille, France}, note = {Data set available at \url{https://github.com/keelm/DIP-SumEval}}, url = {https://aclanthology.org/2020.lrec-1.826}, }

Nermin Avan, Rebecca Zöllner, Eneldo Loza Mencía, Oliver Schellein, and Daniel Kiefl

Evaluation der im Rahmen des ESEG-Projektes implementierten Pflichtelemente für ein EDV-gestütztes Ersteinschätzungssystem in interdisziplinären zentralen Notaufnahmen

In 15. Jahrestagung Deutsche Gesellschaft Interdisziplinäre Notfall- und Akutmedizin (DGINA) , May 2020

Abstract

#public health #outbreak detection #ESEG

Bib URL

@inproceedings{avan20ESEG,
  author = {Avan, Nermin and Z{\"{o}}llner, Rebecca and Loza Menc{\'{i}}a, Eneldo and Schellein, Oliver and Kiefl, Daniel},
  title = {Evaluation der im Rahmen des ESEG-Projektes implementierten Pflichtelemente f{\"{u}}r ein EDV-gest{\"{u}}tztes Ersteinsch{\"{a}}tzungssystem in interdisziplin{\"{a}}ren zentralen Notaufnahmen},
  booktitle = {15. Jahrestagung Deutsche Gesellschaft Interdisziplin{\"{a}}re Notfall- und Akutmedizin (DGINA)},
  year = {2020},
  note = {Abstract},
  pages = {33},
  url = {https://www.dgina-kongress.de/fileadmin/media/dgina/2023/documents/Abstractband_2020_final.pdf},
  url-old = {https://programm.conventus.de/index.php?id=dgina2020&tx\_coprogramm\_programm%5Bprogramm%5D=159&tx\_coprogramm\_programm%5Bsession%5D=73&tx\_coprogramm\_programm%5BcurrentPage%5D=&tx\_coprogramm\_programm%5Baction%5D=programm&tx\_coprogramm\_programm%5Bco},
}

Nermin Avan, Moritz Kulessa, Eneldo Loza Mencía, and Daniel Kiefl

Auswertung der Ersteinschätzung-Routine-Daten unter Verwendung von statistischen Analyseverfahren zur Früherkennung von epidemischen Gefahrenlagen

In 15. Jahrestagung Deutsche Gesellschaft Interdisziplinäre Notfall- und Akutmedizin (DGINA) , May 2020

Abstract

#outbreak detection #syndromic surveillance #public health

Bib URL

@inproceedings{avan20stat,
  author = {Avan, Nermin and Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and Kiefl, Daniel},
  title = {Auswertung der Ersteinsch{\"{a}}tzung-Routine-Daten unter Verwendung von statistischen Analyseverfahren zur Fr{\"{u}}herkennung von epidemischen Gefahrenlagen},
  booktitle = {15. Jahrestagung Deutsche Gesellschaft Interdisziplin{\"{a}}re Notfall- und Akutmedizin (DGINA)},
  year = {2020},
  note = {Abstract},
  pages = {4},
  url = {https://www.dgina-kongress.de/fileadmin/media/dgina/2023/documents/Abstractband_2020_final.pdf},
  url-old = {https://programm.conventus.de/index.php?id=dgina2020&tx\_coprogramm\_programm%5Bprogramm%5D=79&tx\_coprogramm\_programm%5Bsession%5D=28&tx\_coprogramm\_programm%5BcurrentPage%5D=&tx\_coprogramm\_programm%5Baction%5D=programm&tx\_coprogramm\_programm%5Bcon},
}

Michael Rapp, Eneldo Loza Mencía, Johannes Fürnkranz, Vu-Linh Nguyen, and Eyke Hüllermeier

Learning Gradient Boosted Multi-label Classification Rules

In Machine Learning and Knowledge Discovery in Databases (ECML-PKDD) , May 2020

#multi-label #rule-learning #gradient boosting

Abs Bib DOI URL

In multi-label classification, where the evaluation of predic-tions is less straightforward than in single-label classification, variousmeaningful, though different, loss functions have been proposed. Ideally,the learning algorithm should be customizable towards a specific choiceof the performance measure. Modern implementations of boosting, mostprominently gradient boosted decision trees, appear to be appealing fromthis point of view. However, they are mostly limited to single-label clas-sification, and hence not amenable to multi-label losses unless these arelabel-wise decomposable. In this work, we develop a generalization of thegradient boosting framework to multi-output problems and propose analgorithm for learning multi-label classification rules that is able to min-imize decomposable as well as non-decomposable loss functions. Usingthe well-known Hamming loss and subset 0/1 loss as representatives, weanalyze the abilities and limitations of our approach on synthetic dataand evaluate its predictive performance on multi-label benchmarks.
@inproceedings{rapp20boomer, author = {Rapp, Michael and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes and Nguyen, Vu-Linh and H{\"{u}}llermeier, Eyke}, editor = {Hutter, Frank and Kersting, Kristian and Lijffijt, Jefrey and Valera, Isabel}, keywords = {Gradient boosting, multilabel classification, Rule Learning}, title = {Learning Gradient Boosted Multi-label Classification Rules}, booktitle = {Machine Learning and Knowledge Discovery in Databases (ECML-PKDD)}, series = {Lecture Notes in Computer Science}, volume = {12459}, year = {2020}, pages = {124--140}, publisher = {Springer}, url = {https://link.springer.com/chapter/10.1007/978-3-030-67664-3_8}, doi = {https://doi.org/10.1007/978-3-030-67664-3_8}, }
Torsten Dietl, Gjergji Kasneci, Johannes Fürnkranz, and Eneldo Loza Mencía

PLAY: A Profiled Linear Weighting Scheme for Understanding the Influence of Input Variables on the Output of a Deep Artificial Neural Network

Archives of Data Science, Series A, May 2020

Accepted

#interpretable models #deep learning

Bib
@article{td:PLAY, author = {Dietl, Torsten and Kasneci, Gjergji and F{\"{u}}rnkranz, Johannes and Loza Menc{\'{i}}a, Eneldo}, title = {PLAY: A Profiled Linear Weighting Scheme for Understanding the Influence of Input Variables on the Output of a Deep Artificial Neural Network}, journal = {Archives of Data Science, Series A}, year = {2020}, note = {Accepted}, }

2019

Michael Rapp, Eneldo Loza Mencía, and Johannes Fürnkranz

On the Trade-off Between Consistency and Coverage in Multi-label Rule Learning Heuristics

In Discovery Science , Oct 2019

#multi-label #rule-learning #evaluation

Abs Bib DOI URL

Recently, several authors have advocated the use of rule learning algorithms to model multi-label data, as rules are interpretable and can be comprehended, analyzed, or qualitatively evaluated by domain experts. Many rule learning algorithms employ a heuristic-guided search for rules that model regularities contained in the training data and it is commonly accepted that the choice of the heuristic has a significant impact on the predictive performance of the learner. Whereas the properties of rule learning heuristics have been studied in the realm of single-label classification, there is no such work taking into account the particularities of multi-label classification. This is surprising, as the quality of multi-label predictions is usually assessed in terms of a variety of different, potentially competing, performance measures that cannot all be optimized by a single learner at the same time. In this work, we show empirically that it is crucial to trade off the consistency and coverage of rules differently, depending on which multi-label measure should be optimized by a model. Based on these findings, we emphasize the need for configurable learners that can flexibly use different heuristics. As our experiments reveal, the choice of the heuristic is not straight-forward, because a search for rules that optimize a measure locally does usually not result in a model that maximizes that measure globally.
@inproceedings{mr:ML-Consistency-Coverage, author = {Rapp, Michael and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Kralj Novak, Petra and {\v S}muc, Tomislav and D{\v z}eroski, Sa{\v s}o}, keywords = {heuristics, multilabel classification, Rule Learning}, month = oct, title = {On the Trade-off Between Consistency and Coverage in Multi-label Rule Learning Heuristics}, booktitle = {Discovery Science}, year = {2019}, pages = {96--111}, publisher = {Springer International Publishing}, address = {Cham}, isbn = {978-3-030-33778-0}, url = {https://arxiv.org/abs/1908.03032}, doi = {10.1007/978-3-030-33778-0_9}, }
Yannik Klein, Michael Rapp, and Eneldo Loza Mencía

Efficient Discovery of Expressive Multi-label Rules using Relaxed Pruning

In Discovery Science , Oct 2019

Best Student Paper Award

#multi-label #rule-learning

Abs Bib DOI URL

Being able to model correlations between labels is considered crucial in multi-label classification. Rule-based models enable to expose such dependencies, e.g., implications, subsumptions, or exclusions, in an interpretable and human-comprehensible manner. Albeit the number of possible label combinations increases exponentially with the number of available labels, it has been shown that rules with multiple labels in their heads, which are a natural form to model local label dependencies, can be induced efficiently by exploiting certain properties of rule evaluation measures and pruning the label search space accordingly. However, experiments have revealed that multi-label heads are unlikely to be learned by existing methods due to their restrictiveness. To overcome this limitation, we propose a plug-in approach that relaxes the search space pruning used by existing methods in order to introduce a bias towards larger multi-label heads resulting in more expressive rules. We further demonstrate the effectiveness of our approach empirically and show that it does not come with drawbacks in terms of training time or predictive performance.
@inproceedings{yk:Relaxed-Pruning, author = {Klein, Yannik and Rapp, Michael and Loza Menc{\'{i}}a, Eneldo}, editor = {Kralj Novak, Petra and {\v S}muc, Tomislav and D{\v z}eroski, Sa{\v s}o}, keywords = {Label Dependencies, multilabel classification, Rule Learning}, month = oct, title = {Efficient Discovery of Expressive Multi-label Rules using Relaxed Pruning}, booktitle = {Discovery Science}, year = {2019}, pages = {367--382}, publisher = {Springer International Publishing}, note = {Best Student Paper Award}, isbn = {978-3-030-33778-0}, url = {https://arxiv.org/abs/1908.06874}, doi = {10.1007/978-3-030-33778-0_28}, }
Michael Rapp, Eneldo Loza Mencía, and Johannes Fürnkranz

Simplifying Random Forests: On the Trade-off between Interpretability and Accuracy

Sep 2019

Abs Bib URL

We analyze the trade-off between model complexity and accuracy for random forests by breaking the trees up into individual classification rules and selecting a subset of them. We show experimentally that already a few rules are sufficient to achieve an acceptable accuracy close to that of the original model. Moreover, our results indicate that in many cases, this can lead to simpler models that clearly outperform the original ones.
@techreport{mr2019barxiv, author = {Rapp, Michael and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, month = sep, title = {Simplifying Random Forests: On the Trade-off between Interpretability and Accuracy}, type = {ArXiv e-prints}, number = {1911.04393}, year = {2019}, institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt}, url = {https://arxiv.org/abs/1911.04393}, archiveprefix = {arXiv}, eprint = {1911.04393}, primaryclass = {cs.LG} }
Moritz Kulessa, Eneldo Loza Mencía, and Johannes Fürnkranz

Improving Outbreak Detection with Stacking of Statistical Surveillance Methods

In Workshop Proceedings of epiDAMIK: Epidemiology meets Data Mining and Knowledge discovery (held in conjunction with ACM SIGKDD 2019) , Anchorage, USA, Aug 2019

also as arXiv preprint arXiv:1907.07464

#outbreak detection #stacking

Abs Bib URL

Epidemiologists use a variety of statistical algorithms for the early detection of outbreaks. The practical usefulness of such methods highly depends on the trade-off between the detection rate of outbreaks and the chances of raising a false alarm. Recent research has shown that the use of machine learning for the fusion of multiple statistical algorithms improves outbreak detection. Instead of relying only on the binary output (alarm or no alarm) of the statistical algorithms, we propose to make use of their p-values for training a fusion classifier. In addition, we also show that adding additional features and adapting the labeling of an epidemic period may further improve performance. For comparison and evaluation, a new measure is introduced which captures the performance of an outbreak detection method with respect to a low rate of false alarms more precisely than previous works. Our results on synthetic data show that it is challenging to improve the performance with a trainable fusion method based on machine learning. In particular, the use of a fusion classifier that is only based on binary outputs of the statistical surveillance methods can make the overall performance worse than directly using the underlying algorithms. However, the use of p-values and additional information for the learning is promising, enabling to identify more valuable patterns to detect outbreaks.
@inproceedings{mk:epiDAMIK2019, author = {Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, keywords = {Fusion, Outbreak Detection, Stacking, Surveillance}, month = aug, title = {Improving Outbreak Detection with Stacking of Statistical Surveillance Methods}, booktitle = {Workshop Proceedings of epiDAMIK: Epidemiology meets Data Mining and Knowledge discovery (held in conjunction with ACM SIGKDD 2019)}, year = {2019}, location = {Anchorage, USA}, note = {also as arXiv preprint arXiv:1907.07464}, url = {http://people.cs.vt.edu/~badityap/epidamik/kdd-epidamik19-proceedings.pdf}, }

Yannik Klein, Michael Rapp, and Eneldo Loza Mencía

Efficient Discovery of Expressive Multi-label Rules using Relaxed Pruning

Aug 2019

Preprint version of https://ke-tud.github.io/bibtex/publications/show/3174

Bib URL

@techreport{klein19arxiv,
  author = {Klein, Yannik and Rapp, Michael and Loza Menc{\'{i}}a, Eneldo},
  keywords = {Label Dependencies, multilabel classification, Rule Learning},
  title = {Efficient Discovery of Expressive Multi-label Rules using Relaxed Pruning},
  type = {ArXiv e-prints},
  number = {1908.06874},
  year = {2019},
  institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt},
  note = {Preprint version of https://ke-tud.github.io/bibtex/publications/show/3174},
  url = {https://arxiv.org/abs/1908.06874},
  archiveprefix = {arXiv},
  eprint = {1908.06874},
  primaryclass = {cs.LG},
}

Jinseok Nam, Young-Bum Kim, Eneldo Loza Mencía, Sunghyun Park, Ruhi Sarikaya, and Johannes Fürnkranz

Learning Context-dependent Label Permutations for Multi-label Classification

In Proceedings of the 36th International Conference on Machine Learning (ICML-19) , Aug 2019

#multi-label #dynamic classifier chains #extreme classification #scalability #recurrent neural networks #reinforcement learning

Bib URL

@inproceedings{jn:ICML-19,
  author = {Nam, Jinseok and Kim, Young{-}Bum and Loza Menc{\'{i}}a, Eneldo and Park, Sunghyun and Sarikaya, Ruhi and F{\"{u}}rnkranz, Johannes},
  editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
  title = {Learning Context-dependent Label Permutations for Multi-label Classification},
  booktitle = {Proceedings of the 36th International Conference on Machine Learning (ICML-19)},
  series = {Proceedings of Machine Learning Research},
  volume = {97},
  year = {2019},
  pages = {4733--4742},
  publisher = {{PMLR}},
  address = {Long Beach, California, {USA}},
  url = {http://proceedings.mlr.press/v97/nam19a.html},
}

Michael Rapp, Eneldo Loza Mencía, and Johannes Fürnkranz

On the Trade-off Between Consistency and Coverage in Multi-label Rule Learning Heuristics

Aug 2019

Preprint version of https://ke-tud.github.io/bibtex/publications/show/3173

Bib URL

@techreport{mr2019arxiv,
  author = {Rapp, Michael and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  keywords = {heuristics, multilabel classification, Rule Learning},
  title = {On the Trade-off Between Consistency and Coverage in Multi-label Rule Learning Heuristics},
  type = {ArXiv e-prints},
  number = {1908.03032},
  year = {2019},
  institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt},
  note = {Preprint version of https://ke-tud.github.io/bibtex/publications/show/3173},
  url = {https://arxiv.org/abs/1908.03032},
  archiveprefix = {arXiv},
  eprint = {1908.03032},
  primaryclass = {cs.LG}
}

2018

Moritz Kulessa, and Eneldo Loza Mencía

Dynamic Classifier Chain with Random Decision Trees

In Proceedings of the 21st International Conference of Discovery Science (DS-18) , Limassol, Cyprus, Oct 2018

#multi-label #dynamic classifier chains #random decision trees

Abs Bib DOI URL

Classifier chains is an effective approach in order to exploit label dependencies in multi-label data. However, it has the disadvantages that the chain is chosen at total random or relies on a pre-specified ordering of the labels which is expensive to compute. Moreover, the same ordering is used for every test instance, ignoring the fact that different orderings might be best suited for different test instances. We propose a new approach based on random decision trees (RDT) which can choose the label ordering for each prediction dynamically depending on the respective test instance. RDT are not adapted to a specific learning task, but in contrast allow to define a prediction objective on the fly during test time, thus offering a perfect test bed for directly comparing different prediction schemes. Indeed, we show that dynamically selecting the next label improves over using a static ordering of the labels under an otherwise unchanged RDT model and experimental environment.
@inproceedings{mk:DS-18-DCC-RDT, author = {Kulessa, Moritz and Loza Menc{\'{i}}a, Eneldo}, month = oct, title = {Dynamic Classifier Chain with Random Decision Trees}, booktitle = {Proceedings of the 21st International Conference of Discovery Science (DS-18)}, series = {Lecture Notes in Artificial Intelligence}, volume = {11198}, year = {2018}, pages = {33--50}, publisher = {Springer-Verlag}, location = {Limassol, Cyprus}, isbn = {978-3-030-01771-2}, url = {https://ke-tud.github.io/publications/papers/DS18_DynamicClassifierChains_RDT.pdf}, doi = {10.1007/978-3-030-01771-2}, }

Markus Zopf, Teresa Botschen, Tobias Falke, Benjamin Heinzerling, Ana Marasovic, Todor Mihaylov, Avinesh P.V.S., Eneldo Loza Mencía, Johannes Fürnkranz, and Anette Frank

What’s important in a text? An extensive evaluation of linguistic annotations for summarization

In Proceedings of the 5th International Conference on Social Networks Analysis, Management and Security (SNAMS-18) , Valencia, Spain, Oct 2018

#automatic text summarization #feature evaluation

Bib DOI URL

@inproceedings{aiphes:text-important,
  author = {Zopf, Markus and Botschen, Teresa and Falke, Tobias and Heinzerling, Benjamin and Marasovic, Ana and Mihaylov, Todor and P.V.S., Avinesh and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes and Frank, Anette},
  month = oct,
  title = {What’s important in a text? An extensive evaluation of linguistic annotations for summarization},
  booktitle = {Proceedings of the 5th International Conference on Social Networks Analysis, Management and Security (SNAMS-18)},
  year = {2018},
  pages = {272--277},
  location = {Valencia, Spain},
  url = {https://ieeexplore.ieee.org/document/8554853},
  doi = {10.1109/SNAMS.2018.8554853},
}

Patryk Hopner, and Eneldo Loza Mencía

Analysis and Optimization of Deep Counterfactual Value Networks

Jul 2018

Longer version of https://ke-tud.github.io/bibtex/index.php/publications/show/3117

Bib URL

@techreport{hopner18DCVNNlong,
  author = {Hopner, Patryk and Loza Menc{\'{i}}a, Eneldo},
  keywords = {deep neural networks, Game Abstractions, poker},
  month = jul,
  title = {Analysis and Optimization of Deep Counterfactual Value Networks},
  journal = {ArXiv e-prints},
  number = {1807.00900},
  year = {2018},
  institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt},
  note = {Longer version of https://ke-tud.github.io/bibtex/index.php/publications/show/3117},
  url = {https://arxiv.org/abs/1807.00900},
  archiveprefix = {arXiv},
  eprint = {1807.00900},
  primaryclass = {cs.AI}
}

Markus Zopf, Eneldo Loza Mencía, and Johannes Fürnkranz

Which Scores to Predict in Sentence Regression for Text Summarization?

In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018) , Jun 2018

#automatic text summarization #evaluation

Bib DOI URL

@inproceedings{TUD-CS-2018-0032,
  author = {Zopf, Markus and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  editor = {Walker, Marilyn A. and Ji, Heng and Stent, Amanda},
  month = jun,
  title = {Which Scores to Predict in Sentence Regression for Text Summarization?},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018)},
  volume = {1: Long Papers},
  year = {2018},
  pages = {1782--1791},
  url = {http://aclweb.org/anthology/N18-1161},
  doi = {10.18653/v1/N18-1161},
}

Patryk Hopner, and Eneldo Loza Mencía

Analysis and Optimization of Deep Counterfactual Value Networks

In KI 2018: Advances in Artificial Intelligence , Jun 2018

Longer version at https://ke-tud.github.io/bibtex/index.php/publications/show/3118

#poker artificial intelligence #search in game trees #deep learning

Abs Bib DOI URL

Recently a strong poker-playing algorithm called DeepStack was published, which is able to find an approximate Nash equilibrium during gameplay by using heuristic values of future states predicted by deep neural networks. This paper analyzes new ways of encoding the inputs and outputs of DeepStack’s deep counterfactual value networks based on traditional abstraction techniques, as well as an unabstracted encoding, which was able to increase the network’s accuracy.
@inproceedings{hopner18DCVNN, author = {Hopner, Patryk and Loza Menc{\'{i}}a, Eneldo}, editor = {Trollmann, Frank and Turhan, Anni-Yasmin}, keywords = {deep neural networks, Game Abstractions, poker}, title = {Analysis and Optimization of Deep Counterfactual Value Networks}, booktitle = {KI 2018: Advances in Artificial Intelligence}, year = {2018}, pages = {305--312}, publisher = {Springer International Publishing}, note = {Longer version at https://ke-tud.github.io/bibtex/index.php/publications/show/3118}, isbn = {978-3-030-00111-7}, url = {https://ke-tud.github.io/publications/papers/KI18-PokerDCVNN.pdf}, doi = {10.1007/978-3-030-00111-7_26}, }

Eneldo Loza Mencía, Johannes Fürnkranz, Eyke Hüllermeier, and Michael Rapp

Learning Interpretable Rules for Multi-label Classification

In Explainable and Interpretable Models in Computer Vision and Machine Learning , Jun 2018

#multi-label #rule-learning #interpretable models

Bib DOI URL

@incollection{MLRLbook,
  author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes and H{\"{u}}llermeier, Eyke and Rapp, Michael},
  editor = {Jair Escalante, Hugo and Escalera, Sergio and Guyon, Isabelle and Bar{\'{o}}, Xavier and G{\"{u}}{\c c}l{\"{u}}t{\"{u}}rk, Yagmur and G{\"{u}}{\c c}l{\"{u}}, Umut and van Gerven, Marcel A. J.},
  title = {Learning Interpretable Rules for Multi-label Classification},
  booktitle = {Explainable and Interpretable Models in Computer Vision and Machine Learning},
  series = {The Springer Series on Challenges in Machine Learning},
  year = {2018},
  pages = {81--113},
  publisher = {Springer-Verlag},
  isbn = {978-3-319-98131-4},
  url = {https://arxiv.org/abs/1812.00050},
  doi = {https://doi.org/10.1007/978-3-319-98131-4_4},
}

Michael Rapp, Eneldo Loza Mencía, and Johannes Fürnkranz

Exploiting Anti-monotonicity of Multi-label Evaluation Measures for Inducing Multi-label Rules

In PAKDD 2018: Advances in Knowledge Discovery and Data Mining , Jun 2018

#multi-label #rule-learning

Abs Bib DOI URL

Exploiting dependencies between labels is considered to be crucial for multi-label classification. Rules are able to expose label dependencies such as implications, subsumptions or exclusions in a human-comprehensible and interpretable manner. However, the induction of rules with multiple labels in the head is particularly challenging, as the number of label combinations which must be taken into account for each rule grows exponentially with the number of available labels. To overcome this limitation, algorithms for exhaustive rule mining typically use properties such as anti-monotonicity or decomposability in order to prune the search space. In the present paper, we examine whether commonly used multi-label evaluation metrics satisfy these properties and therefore are suited to prune the search space for multi-label heads.
@inproceedings{mr:ML-Antimonotonicity, author = {Rapp, Michael and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Phung, Dinh Q. and Tseng, Vincent S. and Webb, Geoffrey I. and Ho, Bao and Ganji, Mohadeseh and Rashidi, Lida}, title = {Exploiting Anti-monotonicity of Multi-label Evaluation Measures for Inducing Multi-label Rules}, booktitle = {PAKDD 2018: Advances in Knowledge Discovery and Data Mining}, year = {2018}, pages = {29--42}, publisher = {Springer International Publishing}, address = {Cham}, isbn = {978-3-319-93034-3}, url = {https://arxiv.org/abs/1812.06833}, doi = {10.1007/978-3-319-93034-3_3}, }

2017

Camila González, Eneldo Loza Mencía, and Johannes Fürnkranz

Re-training Deep Neural Networks to Facilitate Boolean Concept Extraction

In Proceedings of the 20th International Conference on Discovery Science (DS-17) , Oct 2017

#extraction of rules from neural networks

Bib DOI URL

@inproceedings{jf:DS-17-DNN-Retraining,
  author = {Gonz{\'{a}}lez, Camila and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  month = oct,
  title = {Re-training Deep Neural Networks to Facilitate Boolean Concept Extraction},
  booktitle = {Proceedings of the 20th International Conference on Discovery Science (DS-17)},
  series = {Lecture Notes in Computer Science},
  volume = {10558},
  year = {2017},
  pages = {127--143},
  publisher = {Springer-Verlag},
  isbn = {978-3-319-67785-9},
  url = {https://ke-tud.github.io/publications/papers/DS17-DNN-Retraining.pdf},
  doi = {10.1007/978-3-319-67786-6_10},
}

Jinseok Nam, Eneldo Loza Mencía, Hyunwoo J. Kim, and Johannes Fürnkranz

Maximizing Subset Accuracy with Recurrent Neural Networks in Multi-label Classification

In Advances in Neural Information Processing Systems 30 (NIPS-17) , Oct 2017

#multi-label #extreme classification #scalability #dynamic classifier chains #recurrent neural networks

Abs Bib Code URL

Multi-label classification is the task of predicting a set of labels for a given input instance. Classifier chains are a state-of-the-art method for tackling such problems, which essentially converts this problem into a sequential prediction problem, where the labels are first ordered in an arbitrary fashion, and the task is to predict a sequence of binary values for these labels. In this paper, we replace classifier chains with recurrent neural networks, a sequence-to-sequence prediction algorithm which has recently been successfully applied to sequential prediction tasks in many domains. The key advantage of this approach is that it allows to focus on the prediction of the positive labels only, a much smaller set than the full set of possible labels. Moreover, parameter sharing across all classifiers allows to better exploit information of previous decisions. As both, classifier chains and recurrent neural networks depend on a fixed ordering of the labels, which is typically not part of a multi-label problem specification, we also compare different ways of ordering the label set, and give some recommendations on suitable ordering strategies.
@inproceedings{jn:NIPS-17-MLC-RNN, author = {Nam, Jinseok and Loza Menc{\'{i}}a, Eneldo and Kim, Hyunwoo J. and F{\"{u}}rnkranz, Johannes}, editor = {Guyon, Isabelle and von Luxburg, Ulrike and Bengio, Samy and Wallach, Hanna M. and Fergus, Rob and Vishwanathan, S. V. N. and Garnett, Roman}, title = {Maximizing Subset Accuracy with Recurrent Neural Networks in Multi-label Classification}, booktitle = {Advances in Neural Information Processing Systems 30 (NIPS-17)}, year = {2017}, pages = {5419--5429}, address = {Long Beach, CA}, url = {http://papers.nips.cc/paper/7125-maximizing-subset-accuracy-with-recurrent-neural-networks-in-multi-label-classification}, }

Mohammed Arif Khan, Asif Ekbal, Eneldo Loza Mencía, and Johannes Fürnkranz

Multi-Objective Optimisation-based Feature Selection for Multi-Label Classification

In Proceedings of the 22nd International Conference on Natural Language and Information Systems (NLDB-17) , Oct 2017

#multi-label #feature selection #multi-objective optimization

Bib DOI

@inproceedings{NLDB-17,
  author = {Khan, Mohammed Arif and Ekbal, Asif and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  editor = {Frasincar, Flavius and Ittoo, Ashwin and Nguyen, Le Minh and Metais, Elisabeth},
  title = {Multi-Objective Optimisation-based Feature Selection for Multi-Label Classification},
  booktitle = {Proceedings of the 22nd International Conference on Natural Language and Information Systems (NLDB-17)},
  year = {2017},
  pages = {38--41},
  publisher = {Springer-Verlag},
  address = {Li{\`{e}}ge, Belgium},
  isbn = {978-3-319-59568-9},
  doi = {10.1007/978-3-319-59569-6_5},
}

Mohammed Arif Khan, Asif Ekbal, Eneldo Loza Mencía, and Johannes Fürnkranz

Multi-Objective Optimisation-based Feature Selection for Multi-Label Classification

Oct 2017

Bib URL

@techreport{tr:TUD-KE-2017-01,
  author = {Khan, Mohammed Arif and Ekbal, Asif and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  title = {Multi-Objective Optimisation-based Feature Selection for Multi-Label Classification},
  number = {TUD-KE-2017-01},
  year = {2017},
  institution = {Knowledge Engineeering Group, TU Darmstadt},
  url = {https://ke-tud.github.io/publications/reports/tud-ke-2017-01.pdf}
}

2016

Markus Zopf, Eneldo Loza Mencía, and Johannes Fürnkranz

Sequential Clustering and Contextual Importance Measures for Incremental Update Summarization

In Proceedings of the 26th International Conference on Computational Linguistics , Osaka, Japan, Dec 2016

#automatic text summarization #incremental

Abs Bib URL

Unexpected events such as accidents, natural disasters and terrorist attacks represent an information situation where it is crucial to give users access to important and non-redundant information as early as possible. Previous work uses either a fast but inaccurate pipeline approach or a precise but slow clustering approach. Instead, we propose to use sequential clustering for grouping information so that we are able to publish sentences at each time step. By doing so, we combine the best of both clustering and pipeline approaches and create a fast and precise real-time system. Experiments on the TREC Temporal Summarization 2015 shared task dataset show that our system achieves better results compared to the state-of-the-art.
@inproceedings{ZopfLozaFuernkranz2016IUS, author = {Zopf, Markus and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, month = dec, title = {Sequential Clustering and Contextual Importance Measures for Incremental Update Summarization}, booktitle = {Proceedings of the 26th International Conference on Computational Linguistics}, year = {2016}, pages = {1071-1082}, publisher = {The COLING 2016 Organizing Committee}, location = {Osaka, Japan}, url = {http://aclweb.org/anthology/C16-1102}, }
Markus Zopf, Eneldo Loza Mencía, and Johannes Fürnkranz

Beyond Centrality and Structural Features: Learning Information Importance for Text Summarization

In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning , Berlin, Germany, Aug 2016

#automatic text summarization #pairwise learning

Abs Bib URL

Most automatic text summarization systems proposed to date rely on centrality and structural features as indicators for information importance. In this paper, we argue that these features cannot reliably detect important information in heterogeneous document collections. Instead, we propose CPSum, a summarizer that learns the importance of information objects from a background source. Our hypothesis is tested on a multi-document corpus where we remove centrality and structural features. CPSum proves to be able to perform well in this challenging scenario whereas reference systems fail.
@inproceedings{ZopfLozaFuernkranz2016CPSum, author = {Zopf, Markus and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, month = aug, title = {Beyond Centrality and Structural Features: Learning Information Importance for Text Summarization}, booktitle = {Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning}, year = {2016}, pages = {84-94}, publisher = {Association for Computational Linguistics}, location = {Berlin, Germany}, url = {http://www.aclweb.org/anthology/K16-1009}, }
Eneldo Loza Mencía, Gerard de Melo, and Jinseok Nam

Medical Concept Embeddings via Labeled Background Corpora

In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) , May 2016

#medical embeddings #background corpora

Abs Bib URL

In recent years, we have seen an increasing amount of interest in low-dimensional vector representations of words. Among other things, these facilitate computing word similarity and relatedness scores. The most well-known example of algorithms to produce representations of this sort are the word2vec approaches. In this paper, we investigate a new model to induce such vector spaces for medical concepts, based on a joint objective that exploits not only word co-occurrences but also manually labeled documents, as available from sources such as PubMed. Our extensive experimental analysis shows that our embeddings lead to significantly higher correlations with human similarity and relatedness assessments than previous work. Due to the simplicity and versatility of vector representations, these findings suggest that our resource can easily be used as a drop-in replacement to improve any systems relying on medical concept similarity measures.
@inproceedings{loza16medsim, author = {Loza Menc{\'{i}}a, Eneldo and de Melo, Gerard and Nam, Jinseok}, month = may, title = {Medical Concept Embeddings via Labeled Background Corpora}, booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)}, year = {2016}, pages = {4629--4636}, publisher = {European Language Resources Association (ELRA)}, address = {Paris, France}, isbn = {978-2-9517408-9-1}, url = {http://www.lrec-conf.org/proceedings/lrec2016/pdf/1190_Paper.pdf}, }
Eneldo Loza Mencía, and Frederik Janssen

Learning rules for multi-label classification: a stacking and a separate-and-conquer approach

Machine Learning, May 2016

#multi-label #rule-learning

Abs Bib DOI URL

Dependencies between the labels are commonly regarded as the crucial issue in multi-label classification. Rules provide a natural way for symbolically describing such relationships. For instance, rules with label tests in the body allow for representing directed dependencies like implications, subsumptions, or exclusions. Moreover, rules naturally allow to jointly capture both local and global label dependencies. In this paper, we introduce two approaches for learning such label-dependent rules. Our first solution is a bootstrapped stacking approach which can be built on top of a conventional rule learning algorithm. For this, we learn for each label a separate ruleset, but we include the remaining labels as additional attributes in the training instances. The second approach goes one step further by adapting the commonly used separate-and-conquer algorithm for learning multi-label rules. The main idea is to re-include the covered examples with the predicted labels so that this information can be used for learning subsequent rules. Both approaches allow for making label dependencies explicit in the rules. In addition, the usage of standard rule learning techniques targeted at producing accurate predictions ensures that the found rules are useful for actual classification. Our experiments show (a) that the discovered dependencies contribute to the understanding and improve the analysis of multi-label datasets, and (b) that the found multi-label rules are crucial for the predictive performance as our proposed approaches beat the baseline using conventional rules.
@article{loza16MLRL, author = {Loza Menc{\'{i}}a, Eneldo and Janssen, Frederik}, editor = {D{\v z}eroski, Sa{\v s}o and Kocev, Dragi and Panov, Pan{\v c}e}, keywords = {Label Dependencies, multilabel classification, Rule Learning, Stacking}, month = may, title = {Learning rules for multi-label classification: a stacking and a separate-and-conquer approach}, journal = {Machine Learning}, volume = {105}, number = {1}, year = {2016}, pages = {77--126}, issn = {0885-6125}, url = {https://ke-tud.github.io/publications/papers/loza16MLRL.pdf}, doi = {10.1007/s10994-016-5552-1}, }
Prateek Veeranna Sappadla, Jinseok Nam, Eneldo Loza Mencía, and Johannes Fürnkranz

Using Semantic Similarity for Multi-Label Zero-Shot Classification of Text Documents

In Proceedings of the 23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN-16) , Apr 2016

#multi-label #zero-shot learning #semantic similarity

Abs Bib URL

In this paper, we examine a simple approach to zero-shot multi-label text classification, i.e., to the problem of predicting multiple, possibly previously unseen labels for a document. In particular, we propose to use a semantic embed- ding of label and document words and base the prediction of previously unseen labels on the similarity between the label name and the document words in this em- bedding. Experiments on three textual datasets across various domains show that even such a simple technique yields considerable performance improvements over a simple uninformed baseline.
@inproceedings{esann16zeroshot, author = {Sappadla, Prateek Veeranna and Nam, Jinseok and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, month = apr, title = {Using Semantic Similarity for Multi-Label Zero-Shot Classification of Text Documents}, booktitle = {Proceedings of the 23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN-16)}, year = {2016}, publisher = {d-side publications}, address = {Bruges, Belgium}, url = {https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016-174.pdf}, }

Axel Schulz, Eneldo Loza Mencía, and Benedikt Schmidt

A Rapid-Prototyping Framework for Extracting Small-Scale Incident-Related Information in Microblogs: Application of Multi-Label Classification on Tweets

Information Systems, Apr 2016

#twitter mining #sentiment analysis #multi-label #framework

Bib DOI URL

@article{schulz15mltweets,
  author = {Schulz, Axel and Loza Menc{\'{i}}a, Eneldo and Schmidt, Benedikt},
  month = apr,
  title = {A Rapid-Prototyping Framework for Extracting Small-Scale Incident-Related Information in Microblogs: Application of Multi-Label Classification on Tweets},
  journal = {Information Systems},
  volume = {57},
  year = {2016},
  pages = {88-110},
  url = {https://ke-tud.github.io/publications/papers/infsys16tweetsMLC.pdf},
  doi = {10.1016/j.is.2015.10.010},
}

Jinseok Nam, Eneldo Loza Mencía, and Johannes Fürnkranz

All-in Text: Learning Document, Label, and Word Representations Jointly

In Proceedings of the 30th AAAI Conference on Artificial Intelligence , Phoenix, Arizona, Apr 2016

#multi-label #embeddings #extreme classification #scalability #zero-shot learning

Bib Code URL

@inproceedings{nam16allintext,
  author = {Nam, Jinseok and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  editor = {Schuurmans, Dale and Wellman, Michael P.},
  title = {All-in Text: Learning Document, Label, and Word Representations Jointly},
  booktitle = {Proceedings of the 30th AAAI Conference on Artificial Intelligence},
  year = {2016},
  pages = {1948--1954},
  publisher = {AAAI Press},
  location = {Phoenix, Arizona},
  url = {http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12058},
}

Jan Ruben Zilke, Eneldo Loza Mencía, and Frederik Janssen

DeepRED – Rule Extraction from Deep Neural Networks

In Discovery Science: 19th International Conference, DS 2016, Bari, Italy, October 19–21, 2016, Proceedings , Apr 2016

#extraction of rules from neural networks

Bib DOI URL

@inproceedings{zilke2016deepRED,
  author = {Zilke, Jan Ruben and Loza Menc{\'{i}}a, Eneldo and Janssen, Frederik},
  editor = {Calders, Toon and Ceci, Michelangelo and Malerba, Donato},
  title = {DeepRED -- Rule Extraction from Deep Neural Networks},
  booktitle = {Discovery Science: 19th International Conference, DS 2016, Bari, Italy, October 19--21, 2016, Proceedings},
  year = {2016},
  pages = {457--473},
  publisher = {Springer International Publishing},
  isbn = {978-3-319-46307-0},
  url = {https://ke-tud.github.io/publications/papers/DS16DeepRED.pdf},
  doi = {10.1007/978-3-319-46307-0_29},
}

2015

Mohammed Arif Khan, Asif Ekbal, and Eneldo Loza Mencía

Simultaneous Feature Selection and Parameter Optimization Using Multi-objective Optimization for Sentiment Analysis

In Proceedings of the 12th International Conference on Natural Language Processing (ICON) , Trivandrum, India, Dec 2015

#feature selection #parameter optimization #multi-objective optimization

Bib URL

@inproceedings{arif15moo,
  author = {Khan, Mohammed Arif and Ekbal, Asif and Loza Menc{\'{i}}a, Eneldo},
  month = dec,
  title = {Simultaneous Feature Selection and Parameter Optimization Using Multi-objective Optimization for Sentiment Analysis},
  booktitle = {Proceedings of the 12th International Conference on Natural Language Processing (ICON)},
  year = {2015},
  pages = {285--294},
  publisher = {NLP Association of India},
  location = {Trivandrum, India},
  url = {http://www.aclweb.org/anthology/W15-5942},
}

Jinseok Nam, Eneldo Loza Mencía, Hyunwoo J. Kim, and Johannes Fürnkranz

Predicting Unseen Labels using Label Hierarchies in Large-Scale Multi-label Learning

In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases , Porto, Portugal, Dec 2015

#multi-label #embeddings #zero-shot learning

Abs Bib DOI URL

An important problem in multi-label classification is to capture label patterns or underlying structures that have an impact on such patterns. One way of learning underlying structures over labels is to project both instances and labels into the same space where an instance and its relevant labels tend to have similar representations. In this paper, we present a novel method to learn a joint space of instances and labels by leveraging a hierarchy of labels. We also present an efficient method for pretraining vector representations of labels, namely label embeddings, from large amounts of label co-occurrence patterns and hierarchical structures of labels. This approach also allows us to make predictions on labels that have not been seen during training. We empirically show that the use of pretrained label embeddings allows us to obtain higher accuracies on unseen labels even when the number of labels are quite large. Our experimental results also demonstrate qualitatively that the proposed method is able to learn regularities among labels by exploiting a label hierarchy as well as label co-occurrences.
@inproceedings{nam15wsabieH, author = {Nam, Jinseok and Loza Menc{\'{i}}a, Eneldo and Kim, Hyunwoo J. and F{\"{u}}rnkranz, Johannes}, title = {Predicting Unseen Labels using Label Hierarchies in Large-Scale Multi-label Learning}, booktitle = {Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases}, year = {2015}, pages = {102-118}, publisher = {Springer International Publishing}, location = {Porto, Portugal}, issn = {0302-9743}, isbn = {978-3-319-23527-1}, url = {https://ke-tud.github.io/publications/papers/ECML2015Nam.pdf}, doi = {10.1007/978-3-319-23528-8_7}, }

2014

Christian Brinker, Eneldo Loza Mencía, and Johannes Fürnkranz

Graded Multilabel Classification by Pairwise Comparisons

In 2014 IEEE International Conference on Data Mining (ICDM 2014) , Dec 2014

#multi-label #ordered classification #pairwise learning

Abs Bib DOI URL

The task in multilabel classification is to predict for a given set of labels whether each individual label should be attached to an instance or not. Graded multilabel classification generalizes this setting by allowing to specify for each label a degree of membership on an ordinal scale. This setting can be frequently found in practice, for example when movies or books are assessed on a one-to-five star rating in multiple categories. In this paper, we propose to reformulate the problem in terms of preferences between the labels and their scales, which then be tackled by learning from pairwise comparisons. We present three different approaches which make use of this decomposition and show on three datasets that we are able to outperform baseline approaches. In particular, we show that our solution, which is able to model pairwise preferences across multiple scales, outperforms a straight-forward approach which considers the problem as a set of independent ordinal regression tasks.
@inproceedings{brinker14gmlc, author = {Brinker, Christian and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, month = dec, title = {Graded Multilabel Classification by Pairwise Comparisons}, booktitle = {2014 IEEE International Conference on Data Mining (ICDM 2014)}, year = {2014}, pages = {731--736}, publisher = {Curran Associates, IEEE}, address = {Shenzhen, China}, issn = {1550-4786}, isbn = {978-1-4799-4303-6}, url = {www.ke.tu-darmstadt.de/publications/papers/ICDM14graded.pdf}, doi = {10.1109/ICDM.2014.102}, }
Christian Brinker, Eneldo Loza Mencía, and Johannes Fürnkranz

Graded Multilabel Classification by Pairwise Comparisons

Dec 2014

Longer version of publication at the ICDM 2014, https://ke-tud.github.io/bibtex/publications/show/2618

Abs Bib URL

The task in multilabel classification is to predict for a given set of labels whether each individual label should be attached to an instance or not. Graded multilabel classification generalizes this setting by allowing to specify for each label a degree of membership on an ordinal scale. This setting can be frequently found in practice, for example when movies or books are assessed on a one-to-five star rating in multiple categories. In this paper, we propose to reformulate the problem in terms of preferences between the labels and their scales, which then be tackled by learning from pairwise comparisons. We present three different approaches which make use of this decomposition and show on three datasets that we are able to outperform baseline approaches. In particular, we show that our solution, which is able to model pairwise preferences across multiple scales, outperforms a straight-forward approach which considers the problem as a set of independent ordinal regression tasks.
@techreport{TUD-KE-2014-01, author = {Brinker, Christian and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, month = dec, title = {Graded Multilabel Classification by Pairwise Comparisons}, type = {Technical Report}, year = {2014}, institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt}, note = {Longer version of publication at the ICDM 2014, https://ke-tud.github.io/bibtex/publications/show/2618}, url = {https://ke-tud.github.io/publications/reports/tud-ke-2014-01.pdf}, }

Eneldo Loza Mencía, and Frederik Janssen

Stacking Label Features for Learning Multilabel Rules

In Discovery Science - 17th International Conference, DS 2014, Bled, Slovenia, October 8-10, 2014, Proceedings , Oct 2014

#multi-label #rule-learning #stacking

Bib DOI URL

@incollection{loza14MLRL,
  author = {Loza Menc{\'{i}}a, Eneldo and Janssen, Frederik},
  editor = {D{\v z}eroski, Sa{\v s}o and Panov, Pan{\v c}e and Kocev, Dragi and Todorovski, Ljup{\v c}o},
  month = oct,
  title = {Stacking Label Features for Learning Multilabel Rules},
  booktitle = {Discovery Science - 17th International Conference, DS 2014, Bled, Slovenia, October 8-10, 2014, Proceedings},
  series = {Lecture Notes in Computer Science},
  volume = {8777},
  year = {2014},
  pages = {192-203},
  publisher = {Springer},
  isbn = {978-3-319-11811-6},
  url = {https://ke-tud.github.io/publications/papers/ds14stackinglabelfeatures.pdf},
  doi = {10.1007/978-3-319-11812-3_17},
}

Jinseok Nam, Jungi Kim, Eneldo Loza Mencía, Iryna Gurevych, and Johannes Fürnkranz

Large-Scale Multi-label Text Classification - Revisiting Neural Networks

In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD-14), Part 2 , Sep 2014

#multi-label #neural networks #deep learning

Bib DOI URL

@inproceedings{nam14revisiting,
  author = {Nam, Jinseok and Kim, Jungi and Loza Menc{\'{i}}a, Eneldo and Gurevych, Iryna and F{\"{u}}rnkranz, Johannes},
  editor = {Calders, Toon and Esposito, Floriana and H{\"{u}}llermeier, Eyke and Meo, Rosa},
  month = sep,
  title = {Large-Scale Multi-label Text Classification - Revisiting Neural Networks},
  booktitle = {Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD-14), Part 2},
  series = {Lecture Notes in Computer Science},
  volume = {8725},
  year = {2014},
  pages = {437--452},
  publisher = {Springer Berlin Heidelberg},
  isbn = {978-3-662-44850-2},
  url = {https://ke-tud.github.io/publications/papers/NN_MLC_ecml2014_camera_ready.pdf},
  doi = {10.1007/978-3-662-44851-9_28},
}

Petar Ristoski, Eneldo Loza Mencía, and Heiko Paulheim

A Hybrid Multi-Strategy Recommender System Using Linked Open Data

In Semantic Web Evaluation Challenge, Proceedings (ESWC 2014) , May 2014

#recommender systems #linked open data #multi-label #competition

Abs Bib DOI URL

In this paper, we discuss the development of a hybrid multi-strategy book recommendation system using Linked Open Data. Our approach builds on training individual base recommenders and using global popularity scores as generic recommenders. The results of the individual recommenders are combined using stacking regression and rank aggregation. We show that this approach delivers very good results in different recommendation settings and also allows for incorporating diversity of recommendations.
@incollection{loza14recsys, author = {Ristoski, Petar and Loza Menc{\'{i}}a, Eneldo and Paulheim, Heiko}, month = may, title = {A Hybrid Multi-Strategy Recommender System Using Linked Open Data}, booktitle = {Semantic Web Evaluation Challenge, Proceedings (ESWC 2014)}, series = {Communications in Computer and Information Science}, volume = {475}, year = {2014}, pages = {150-156}, publisher = {Springer}, isbn = {978-3-319-12023-2}, url = {http://2014.eswc-conferences.org/sites/default/files/eswc2014-challenges_rs_submission_12.pdf}, doi = {10.1007/978-3-319-12024-9_19}, }

Axel Schulz, Eneldo Loza Mencía, Thanh Tung Dang, and Benedikt Schmidt

Evaluating Multi-label Classification of Incident-related Tweets

In Proceedings, 4th Workshop on Making Sense of Microposts (#Microposts2014) at WWW: Big things come in small packages , Apr 2014

#twitter mining #multi-label

Bib URL

@inproceedings{loza14twitterml,
  author = {Schulz, Axel and Loza Menc{\'{i}}a, Eneldo and Dang, Thanh Tung and Schmidt, Benedikt},
  keywords = {Microblogs, multi-label learning, Social Media},
  month = apr,
  title = {Evaluating Multi-label Classification of Incident-related Tweets},
  booktitle = {Proceedings, 4th Workshop on Making Sense of Microposts {(\#Microposts2014)} at WWW: Big things come in small packages},
  year = {2014},
  pages = {26--33},
  url = {http://ceur-ws.org/Vol-1141/paper_01.pdf},
}

2013

Eneldo Loza Mencía, Jinseok Nam, and Dong-Hyun Lee

Learning multi-labeled bioacoustic samples with an unsupervised feature learning approach

In Proceedings of Neural Information Scaled for Bioacoustics, from Neurons to Big Data , Dec 2013

Proceedings of NIPS4B workshop joint to NIPS

#multi-label #audio analysis #deep learning #competition

Abs Bib URL

Multi-label Bird Species Classification competition provides an excellent opportunity to analyze the effectiveness of acoustic processing and mutlilabel learning. We propose an unsupervised feature extraction and generation approach based on latest advances in deep neural network learning, which can be applied generically to acoustic data. With state-of-the-art approaches from multilabel learning, we achieved top positions in the competition, only surpassed by teams with profound expertise in acoustic data processing.
@inproceedings{lozanam2013nips4b, author = {Loza Menc{\'{i}}a, Eneldo and Nam, Jinseok and Lee, Dong-Hyun}, editor = {Glotin, H. and LeCun, Yann and Mallat, St{\'{e}}phane and Tchernichovski, Ofer and Arti{\`{e}}res, Thierry and Halkias, Xanadu}, month = dec, title = {Learning multi-labeled bioacoustic samples with an unsupervised feature learning approach}, booktitle = {Proceedings of Neural Information Scaled for Bioacoustics, from Neurons to Big Data}, year = {2013}, pages = {184-189}, organization = {NIPS Int. Conf.}, note = {Proceedings of NIPS4B workshop joint to NIPS}, issn = {979-10-90821-04-0}, url = {https://ke-tud.github.io/publications/papers/lozanam2013nips4b.pdf}, }

Eneldo Loza Mencía, Simon Holthausen, Axel Schulz, and Frederik Janssen

Using Data Mining on Linked Open Data for Analyzing E-Procurement Information

In Proceedings of the first DMoLD: Data Mining on Linked Data Workshop at ECML/PKDD2013 , Dec 2013

#linked open data #cost-sensitive learning #competition

Bib URL

@inproceedings{loza13DMoLD,
  author = {Loza Menc{\'{i}}a, Eneldo and Holthausen, Simon and Schulz, Axel and Janssen, Frederik},
  title = {Using Data Mining on Linked Open Data for Analyzing E-Procurement Information},
  booktitle = {Proceedings of the first DMoLD: Data Mining on Linked Data Workshop at ECML/PKDD2013},
  year = {2013},
  url = {http://ceur-ws.org/Vol-1082/paper4.pdf},
}

Eneldo Loza Mencía, and Frederik Janssen

Towards Multilabel Rule Learning

In Proceedings of the German Workshop on Lernen, Wissen, Adaptivität - LWA2013 , Dec 2013

#multi-label #rule-learning #position paper

Bib URL

@inproceedings{loza13MLRL,
  author = {Loza Menc{\'{i}}a, Eneldo and Janssen, Frederik},
  editor = {Henrich, Andreas and Sperker, Hans-Christian},
  title = {Towards Multilabel Rule Learning},
  booktitle = {Proceedings of the German Workshop on Lernen, Wissen, Adaptivit{\"{a}}t - LWA2013},
  year = {2013},
  pages = {155--158},
  address = {Bamberg, Germany},
  url = {http://www.minf.uni-bamberg.de/lwa2013/proceedings/proceedings_lwa1013.pdf},
}

2012

Wouter Duivesteijn, Eneldo Loza Mencía, Johannes Fürnkranz, and Arno J. Knobbe

Multi-label LeGo – Enhancing Multi-label Classifiers with Local Patterns

In Advances in Intelligent Data Analysis XI – Proceedings of the 11th International Symposium on Data Analysis (IDA-11) , Berlin, Oct 2012

Longer version available at https://ke-tud.github.io/bibtex/publications/show/2342

#multi-label #local patterns #exceptionality mining

Bib DOI URL

@inproceedings{duivesteijn12MultilabelLego,
  author = {Duivesteijn, Wouter and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes and Knobbe, Arno J.},
  editor = {Hollm{\'{e}}n, Jaakko and Klawonn, Frank and Tucker, Allan},
  month = oct,
  title = {Multi-label LeGo -- Enhancing Multi-label Classifiers with Local Patterns},
  booktitle = {Advances in Intelligent Data Analysis XI -- Proceedings of the 11th International Symposium on Data Analysis (IDA-11)},
  series = {Lecture Notes in Computer Science},
  volume = {7619},
  year = {2012},
  pages = {114--125},
  publisher = {Springer},
  location = {Berlin},
  note = {Longer version available at https://ke-tud.github.io/bibtex/publications/show/2342},
  isbn = {978-3-642-34155-7},
  url = {https://ke-tud.github.io/publications/papers/IDA-12.pdf},
  doi = {10.1007/978-3-642-34156-4_12},
}

Eneldo Loza Mencía

Efficient Pairwise Multilabel Classification

Jul 2012

submitted on 2012-06-10, defended on 2012-07-24

Abs Bib URL

Multilabel classification learning is the task of learning a mapping between objects and sets of possibly overlapping classes and has gained increasing attention in recent times. A prototypical application scenario for multilabel classification is the assignment of a set of keywords to a document, a frequently encountered problem in the text classification domain. With upcoming Web 2.0 technologies, this domain is extended by a wide range of tag suggestion tasks and the trend definitely is moving towards more data points and more labels. This work provides an extended introduction into the topic of multilabel classification, a detailed formalization and a comprehensive overview of the present state-of-the-art approaches. A commonly used solution for solving multilabel tasks is to decompose the original problem into several subproblems. These subtasks are usually easy to solve with conventional techniques. In contrast to the straightforward approach of training one classifier for independently predicting the relevance of each class (binary relevance), this work focuses particularly on the pairwise decomposition of the original problem in which a decision function is learned for each possible pair of classes. The main advantage of this approach, the improvement of the predictive quality, comes at the cost of its main disadvantage, the quadratic number of classifiers needed (with respect to the number of labels). This thesis presents a framework of efficient and scalable solutions for handling hundreds or thousands of labels despite the quadratic dependency. As it turns out, training such a pairwise ensemble of classifiers can be accomplished in linear time and only differs from the straightforward binary relevance approach (BR) by a factor relative to the average number of labels associated to an object, which is usually small. Furthermore, the integration of a smart scheduling technique inspired from sports tournaments safely reduces the quadratic number of base classifier evaluations to log-linear in practice. Combined with a simple yet fast and powerful learning algorithm for linear classifiers, data with a huge number of high dimensional points, which was not amenable to pairwise learning before, can be processed even under real-time conditions. The remaining bottleneck, the exploding memory requirements, is coped by taking advantage of an interesting property of linear classifiers, namely the possibility of dual reformulation as a linear combination of the training examples. The suitability is demonstrated on the novel EUR-Lex text collection, which particularly puts the main scalability issue of pairwise learning to test. With its almost 4,000 labels and 20,000 documents it is one of the most challenging test beds in multilabel learning to date. The dual formulation allows to maintain the mathematical equivalent to 8 million base learners needed for conventionally solving EUR-Lex in almost the same amount of space as binary relevance. Moreover, BR was clearly beaten in the experiments. A further contribution based on hierarchical decomposition and arrangement of the original problem allows to reduce the dependency on the number of labels to even sub-linearity. This approach opens the door to a wide range of new challenges and applications but simultaneously maintains the advantages of pairwise learning, namely the excellent predictive quality. It was even shown in comparison to the flat variant that it has a particularly positive effect on balancing recall and precision on datasets with a large number of labels. The improved scalability and efficiency allowed to apply pairwise classification to a set of large multilabel problems with a parallel base of data points but different domains of labels. A first attempt was made in this parallel tasks setting in order to investigate the exploitation of label dependencies by pairwise learning, with first encouraging results. The usage of multilabel learning techniques for the automatic annotation of texts constitutes a further obvious but so far missing connection to multi-task and multi-target learning. The presented solution considers the simultaneous tagging of words with different but possibly overlapping annotation schemes as a multilabel problem. This solution is expected to particularly benefit from approaches which exploit label dependencies. The ability of pairwise learning for this purpose is obviously restricted to pairwise relations, therefore a technique is investigated which explores label constellations that only exist locally for a subgroup of data points. In addition to the positive effect of the supplemental information, the experimental evaluation demonstrates an interesting insight with regards to the different behavior of several state-of-the-art approaches with respect to the optimization of particular multilabel measures, a controversial topic in multilabel classification.
@phdthesis{loza2012diss, author = {Loza Menc{\'{i}}a, Eneldo}, keywords = {efficiency, multilabel classification, pairwise classification, scalability}, month = jul, title = {Efficient Pairwise Multilabel Classification}, type = {Dissertation}, year = {2012}, school = {Technische Universit{\"{a}}t Darmstadt, Knowledge Engineering Group}, note = {submitted on 2012-06-10, defended on 2012-07-24}, url = {http://tuprints.ulb.tu-darmstadt.de/3226/7/loza12diss.pdf}, urn = {urn:nbn:de:tuda-tuprints-32260}, tuprints-url = {http://tuprints.ulb.tu-darmstadt.de/3226/} }

Wouter Duivesteijn, Eneldo Loza Mencía, Johannes Fürnkranz, and Arno J. Knobbe

Multi-label LeGo – Enhancing Multi-label Classifiers with Local Patterns

Jul 2012

Longer version of publication at the IDA 2012, https://ke-tud.github.io/bibtex/publications/show/2303

Bib URL

@techreport{TUD-KE-2012-02,
  author = {Duivesteijn, Wouter and Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes and Knobbe, Arno J.},
  title = {Multi-label LeGo -- Enhancing Multi-label Classifiers with Local Patterns},
  number = {TUD-KE-2012-02},
  year = {2012},
  institution = {Knowledge Engineering Group, Technische Universit{\"{a}}t Darmstadt},
  note = {Longer version of publication at the IDA 2012, https://ke-tud.github.io/bibtex/publications/show/2303},
  url = {https://ke-tud.github.io/publications/reports/tud-ke-2012-02.pdf}
}

2010

Eneldo Loza Mencía

An Evaluation of Multilabel Classification for the Automatic Annotation of Texts

In Proceedings of the LWA 2010: Lernen, Wissen, Adaptivität, Workshop on Knowledge Discovery, Data Mining and Machine Learning (KDML 2010) , Kassel, Oct 2010

#syntactic parsing #multi-label

Abs Bib URL

This article presents the formulation of an information extraction as a multilabel classification problem. This representation allows for exploiting annotation overlappings and correlations. The standard multiclass approach is compared to different multilabel classification algorithms.
@inproceedings{loza10mlie, author = {Loza Menc{\'{i}}a, Eneldo}, editor = {Atzm{\"{u}}ller, Martin and Benz, Dominik and Hotho, Andreas and Stumme, Gerd}, month = oct, title = {An Evaluation of Multilabel Classification for the Automatic Annotation of Texts}, booktitle = {Proceedings of the LWA 2010: Lernen, Wissen, Adaptivit{\"{a}}t, Workshop on Knowledge Discovery, Data Mining and Machine Learning (KDML 2010)}, year = {2010}, pages = {121-123}, location = {Kassel}, url = {http://www.kde.cs.uni-kassel.de/conf/lwa10/papers/kdml11.pdf}, }
Eneldo Loza Mencía

Multilabel Classification in Parallel Tasks

In Working Notes of the 2nd International Workshop on Learning from Multi-Label Data at ICML/COLT 2010 , Haifa, Israel, Jun 2010

#multi-label #transfer learning

Abs Bib URL

In real world multilabel problems, it is often the case that e.g. documents are simultaneously classified with labels from multiple domains, such as genres in addition to topics. In practice, each of these problems is solved independently without taking advantage of possible label correlations between domains. Following the multi-task learning setting, in which multiple similar tasks are learned in parallel, we propose a global learning approach that jointly considers all domains. It is empirically demonstrated in this work that this approach is effective despite its simplicity when using a multilabel learner that takes label correlations into account.
@inproceedings{loza10pt, author = {Loza Menc{\'{i}}a, Eneldo}, editor = {Zhang, Min-Ling and Tsoumakas, Grigorios and Zhou, Zhi-Hua}, month = jun, title = {Multilabel Classification in Parallel Tasks}, booktitle = {Working Notes of the 2nd International Workshop on Learning from Multi-Label Data at ICML/COLT 2010}, year = {2010}, pages = {29-36}, location = {Haifa, Israel}, url = {https://ke-tud.github.io/publications/papers/loza10mlpt.pdf}, crossref = {zhang10mlworkshop}, }
Eneldo Loza Mencía, and Johannes Fürnkranz

Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain

In Semantic Processing of Legal Texts – Where the Language of Law Meets the Law of Language , May 2010

accompanying EUR-Lex dataset available at \urlhttps://ke-tud.github.io/resources/eurlex

#multi-label #rule-learning #scalability #data set

Abs Bib DOI URL

In this paper we apply multilabel classification algorithms to the EUR-Lex database of legal documents of the European Union. For this document collection, we studied three different multilabel classification problems, the largest being the categorization into the EUROVOC concept hierarchy with almost 4000 classes. We evaluated three algorithms: (i) the binary relevance approach which independently trains one classifier per label; (ii) the multiclass multilabel perceptron algorithm, which respects dependencies between the base classifiers; and (iii) the multilabel pairwise perceptron algorithm, which trains one classifier for each pair of labels. All algorithms use the simple but very efficient perceptron algorithm as the underlying classifier, which makes them very suitable for large-scale multilabel classification problems. The main challenge we had to face was that the almost 8,000,000 perceptrons that had to be trained in the pairwise setting could no longer be stored in memory. We solve this problem by resorting to the dual representation of the perceptron, which makes the pairwise approach feasible for problems of this size. The results on the EUR-Lex database confirm the good predictive performance of the pairwise approach and demonstrates the feasibility of this approach for large-scale tasks.
@incollection{jf:SemanticLaw, author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Francesconi, Enrico and Montemagni, Simonetta and Peters, Wim and Tiscornia, Daniela}, keywords = {EUR-Lex Database, learning by pairwise comparison, Legal Documents, multilabel classification, Text Classification}, month = may, title = {Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain}, booktitle = {Semantic Processing of Legal Texts -- Where the Language of Law Meets the Law of Language}, edition = {1}, series = {Lecture Notes in Artificial Intelligence}, volume = {6036}, year = {2010}, pages = {192-215}, publisher = {Springer-Verlag}, note = {accompanying EUR-Lex dataset available at \url{https://ke-tud.github.io/resources/eurlex}}, isbn = {978-3-642-12836-3}, url = {https://ke-tud.github.io/publications/papers/loza10eurlex.pdf}, doi = {10.1007/978-3-642-12837-0_11}, }
Eneldo Loza Mencía, Sang-Hyeun Park, and Johannes Fürnkranz

Efficient Voting Prediction for Pairwise Multilabel Classification

Neurocomputing, Mar 2010

#multi-label #pairwise learning #aggregation #scalability

Abs Bib DOI URL

The pairwise approach to multilabel classification reduces the problem to learning and aggregating preference predictions among the possible labels. A key problem is the need to query a quadratic number of preferences for making a prediction. To solve this problem, we extend the recently proposed QWeighted algorithm for efficient pairwise multiclass voting to the multilabel setting, and evaluate the adapted algorithm on several real-world datasets. We achieve an average-case reduction of classifier evaluations from n^2 to n + n d log n, where n is the total number of possible labels and d is the average number of labels per instance, which is typically quite small in real-world datasets.
@article{jf:Neurocomputing, author = {Loza Menc{\'{i}}a, Eneldo and Park, Sang-Hyeun and F{\"{u}}rnkranz, Johannes}, keywords = {efficient classification, learning by pairwise comparison, multilabel classification, voting aggregation}, month = mar, title = {Efficient Voting Prediction for Pairwise Multilabel Classification}, journal = {Neurocomputing}, volume = {73}, number = {7-9}, year = {2010}, pages = {1164 - 1176}, issn = {0925-2312}, url = {https://ke-tud.github.io/publications/papers/neucom10.pdf}, doi = {10.1016/j.neucom.2009.11.024}, note2 = {Volume: Advances in Computational Intelligence and Learning - 17th European Symposium on Artificial Neural Networks 2009, 17th European Symposium on Artificial Neural Networks 2009}, }

2009

Eneldo Loza Mencía, Sang-Hyeun Park, and Johannes Fürnkranz

Efficient Voting Prediction for Pairwise Multilabel Classification

In Proceedings of the LWA 2009: Lernen - Wissen - Adaption, Workshop Knowledge Discovery, Data Mining and Machine Learning (KDML-09) , Darmstadt, Germany, Aug 2009

resubmission

#multi-label #pairwise learning #aggregation #scalability

Bib URL

@inproceedings{jf:LWA-09-Pairwise,
  author = {Loza Menc{\'{i}}a, Eneldo and Park, Sang-Hyeun and F{\"{u}}rnkranz, Johannes},
  editor = {Benz, Dominik and Janssen, Frederik},
  month = aug,
  title = {Efficient Voting Prediction for Pairwise Multilabel Classification},
  booktitle = {Proceedings of the LWA 2009: Lernen - Wissen - Adaption, Workshop Knowledge Discovery, Data Mining and Machine Learning (KDML-09)},
  year = {2009},
  pages = {72--75},
  location = {Darmstadt, Germany},
  address = {Darmstadt, Germany},
  note = {resubmission},
  url = {https://ke-tud.github.io/publications/papers/KDML09.pdf},
  crossref = {LWA2009},
}

Eneldo Loza Mencía

Segmentation of legal documents

In Proceedings of the 12th International Conference on Artificial Intelligence and Law , Barcelona, Spain, Jun 2009

#text segmentation #information extraction

Abs Bib DOI URL

An overwhelming number of legal documents is available in digital form. However, most of the texts are usually only provided in a semi-structured form, i.e. the documents are structured only implicitly using text formatting and alignment. In this form the documents are perfectly understandable by a human, but not by a machine. This is an obstacle towards advanced intelligent legal information retrieval and knowledge systems. The reason for this lack of structured knowledge is that the conversion of texts in conventional form into a structured, machine-readable form, a process called segmentation, is frequently done manually and is therefore very expensive. We introduce a trainable system based on state-of-the-art Information Extraction techniques for the automatic segmentation of legal documents. Our system makes special use of the implicitly given structure in the source digital file as well as of the explicit knowledge about the target structure. Our evaluation on the French IPR Law demonstrates that the system is able to learn an effective segmenter given only a few manually processed training documents. In some cases, even only one seen example is sufficient in order to correctly process the remaining documents.
@inproceedings{loza09segmentation, author = {Loza Menc{\'{i}}a, Eneldo}, month = jun, title = {Segmentation of legal documents}, booktitle = {Proceedings of the 12th International Conference on Artificial Intelligence and Law}, year = {2009}, pages = {88--97}, publisher = {ACM}, location = {Barcelona, Spain}, address = {New York, NY, USA}, isbn = {978-1-60558-597-0}, url = {https://ke-tud.github.io/publications/papers/loza09segmentation.pdf}, doi = {10.1145/1568234.1568245}, }
Eneldo Loza Mencía, Sang-Hyeun Park, and Johannes Fürnkranz

Efficient Voting Prediction for Pairwise Multilabel Classification

In Proceedings of the 17th European Symposium on Artificial Neural Networks (ESANN 2009, Bruges, Belgium) , Apr 2009

#multi-label #pairwise learning

Abs Bib URL

The pairwise approach to multilabel classification reduces the problem to learning and aggregating preference predictions among the possible labels. A key problem is the need to query a quadratic number of preferences for making a prediction. To solve this problem, we extend the recently proposed QWeighted algorithm for efficient pairwise multiclass voting to the multilabel setting, and evaluate the adapted algorithm on several real-world datasets. We achieve an average-case reduction of classifier evaluations from n^2 to n + dn log n, where n is the total number of labels and d is the average number of labels, which is typically quite small in real-world datasets.
@inproceedings{jf:ESANN-09, author = {Loza Menc{\'{i}}a, Eneldo and Park, Sang-Hyeun and F{\"{u}}rnkranz, Johannes}, month = apr, title = {Efficient Voting Prediction for Pairwise Multilabel Classification}, booktitle = {Proceedings of the 17th European Symposium on Artificial Neural Networks (ESANN 2009, Bruges, Belgium)}, year = {2009}, pages = {117--122}, publisher = {d-side publications}, isbn = {2-930307-09-9}, url = {http://www.dice.ucl.ac.be/Proceedings/esann/esannpdf/es2009-112.pdf}, opturl = {"https://ke-tud.github.io/publications/papers/ESANN09.pdf"}, }
Grigorios Tsoumakas, Eneldo Loza Mencía, Ioannis Katakis, Sang-Hyeun Park, and Johannes Fürnkranz

On the Combination of Two Decompositive Multi-Label Classification Methods

In Proceedings of the ECML PKDD 2009 Workshop on Preference Learning (PL-09, Bled, Slovenia) , Apr 2009

#multi-label #pairwise learning #scalability

Abs Bib URL

In this paper, we compare and combine two approaches for multi-label classification that both decompose the initial problem into sets of smaller problems. The Calibrated Label Ranking approach is based on interpreting the multi-label problem as a preference learning problem and decomposes it into a quadratic number of binary classifiers. The HOMER approach reduces the original problem into a hierarchy of considerably simpler multi-label problems. Experimental results indicate that the use of HOMER is beneficial for the pairwise preference-based approach in terms of computational cost and quality of prediction.
@inproceedings{jf:PL-09-WS-Paper, author = {Tsoumakas, Grigorios and Loza Menc{\'{i}}a, Eneldo and Katakis, Ioannis and Park, Sang-Hyeun and F{\"{u}}rnkranz, Johannes}, editor = {H{\"{u}}llermeier, Eyke and F{\"{u}}rnkranz, Johannes}, title = {On the Combination of Two Decompositive Multi-Label Classification Methods}, booktitle = {Proceedings of the ECML PKDD 2009 Workshop on Preference Learning (PL-09, Bled, Slovenia)}, year = {2009}, pages = {114--129}, url = {https://ke-tud.github.io/events/PL-09/09-Tsoumakas.pdf}, }

2008

Eneldo Loza Mencía, and Johannes Fürnkranz

Mapping of Documents and Segmentation

Sep 2008

Project Reference 027968

Bib

@misc{alisD55,
  author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes},
  month = sep,
  title = {Mapping of Documents and Segmentation},
  year = {2008},
  howpublished = {ALIS Project Deliverable {WP5/D5.5} --- IST-2004-2.4.9},
  note = {Project Reference 027968}
}

Eneldo Loza Mencía, and Johannes Fürnkranz

Efficient Pairwise Multilabel Classification for Large-Scale Problems in the Legal Domain

In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Disocvery in Databases (ECML-PKDD-2008), Part II , Sep 2008

accompanying EUR-Lex dataset available at \urlhttps://ke-tud.github.io/resources/eurlex

#multi-label #pairwise learning #scalability #data set

Abs Bib DOI URL

In this paper we applied multilabel classification algorithms to the EUR-Lex database of legal documents of the European Union. On this document collection, we studied three different multilabel classification problems, the largest being the categorization into the EUROVOC concept hierarchy with almost 4000 classes. We evaluated three algorithms: (i) the binary relevance approach which independently trains one classifier per label; (ii) the multiclass multilabel perceptron algorithm, which respects dependencies between the base classifiers; and (iii) the multilabel pairwise perceptron algorithm, which trains one classifier for each pair of labels. All algorithms use the simple but very efficient perceptron algorithm as the underlying classifier, which makes them very suitable for large-scale multilabel classification problems. The main challenge we had to face was that the almost 8,000,000 perceptrons that had to be trained in the pairwise setting could no longer be stored in memory. We solve this problem by resorting to the dual representation of the perceptron, which makes the pairwise approach feasible for problems of this size. The results on the EUR-Lex database confirm the good predictive performance of the pairwise approach and demonstrates the feasibility of this approach for large-scale tasks.
@inproceedings{jf:ECML-PKDD-08, author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Daelemans, Walter and Goethals, Bart and Morik, Katharina}, month = sep, title = {Efficient Pairwise Multilabel Classification for Large-Scale Problems in the Legal Domain}, booktitle = {Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Disocvery in Databases (ECML-PKDD-2008), Part II}, series = {Lecture Notes in Computer Science}, volume = {5212}, year = {2008}, pages = {50--65}, publisher = {Springer}, address = {Antwerp, Belgium}, note = {accompanying EUR-Lex dataset available at \url{https://ke-tud.github.io/resources/eurlex}}, isbn = {978-3-540-87480-5}, url = {https://ke-tud.github.io/publications/papers/ECML08.pdf}, doi = {10.1007/978-3-540-87481-2_4}, }

Johannes Fürnkranz, Eneldo Loza Mencía, Heiko Paulheim, Hélène Bestougeff, Migle Laukyte, Rossella Rubino, Giuseppe Contissa, LLoyd Kamara, Amar-Djalil Mezaour, Daniel Ramirez, Régis Riveret, and Michel Rudnianski

Report on ALIS Ontology

Jun 2008

Project Reference 027968

Bib

@misc{alisD54,
  author = {F{\"{u}}rnkranz, Johannes and Loza Menc{\'{i}}a, Eneldo and Paulheim, Heiko and Bestougeff, H{\'{e}}l{\`{e}}ne and Laukyte, Migle and Rubino, Rossella and Contissa, Giuseppe and Kamara, LLoyd and Mezaour, Amar-Djalil and Ramirez, Daniel and Riveret, R{\'{e}}gis and Rudnianski, Michel},
  month = jun,
  title = {Report on ALIS Ontology},
  year = {2008},
  howpublished = {ALIS Project Deliverable {WP5/D5.4} --- IST-2004-2.4.9},
  note = {Project Reference 027968}
}

Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Klaus Brinker

Multilabel Classification via Calibrated Label Ranking

Machine Learning, Jun 2008

#multi-label #pairwise learning

Bib DOI URL

@article{jf:MLJ-Multilabel,
  author = {F{\"{u}}rnkranz, Johannes and H{\"{u}}llermeier, Eyke and Loza Menc{\'{i}}a, Eneldo and Brinker, Klaus},
  month = jun,
  title = {Multilabel Classification via Calibrated Label Ranking},
  journal = {Machine Learning},
  volume = {73},
  number = {2},
  year = {2008},
  pages = {133--153},
  url = {https://ke-tud.github.io/publications/papers/MLJ08.pdf},
  doi = {10.1007/s10994-008-5064-8},
}

Eneldo Loza Mencía, and Johannes Fürnkranz

An Evaluation of Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain

In Proceedings of the LREC 2008 Workshop on Semantic Processing of Legal Texts , May 2008

#multi-label #scalability

Abs Bib URL

In this paper we evaluate the performance of multilabel classification algorithms on the EUR-Lex database of legal documents of the European Union. On the same set of underlying documents, we defined three different large-scale multilabel problems with up to 4000 classes. On these datasets, we compared three algorithms: (i) the well-known one-against-all approach (OAA); (ii) the multiclass multilabel perceptron algorithm (MMP), which modifies the OAA ensemble by respecting dependencies between the base classifiers in the training protocol of the classifier ensemble; and (iii) the multilabel pairwise perceptron algorithm (MLPP), which unlike the previous algorithms trains one base classifier for each pair of classes. All algorithms use the simple but very efficient perceptron algorithm as the underlying classifier. This makes them very suitable for large-scale multilabel classification problems. While previous work has already shown that the latter approach outperforms the other two approaches in terms of predictive accuracy, its key problem is that it has to store one classifier for each pair of classes. The key contribution of this work is to demonstrate a novel technique that makes the pairwise approach feasible for problems with large number of classes, such as those studied in this work. Our results on the EUR-Lex database illustrate the effectiveness of the pairwise approach and the efficiency of the MMP algorithm. We also show that it is feasible to efficiently and effectively handle very large multilabel problems.
@inproceedings{jf:LREC-08-WS, author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Montemagni, Simonetta and Tiscornia, Daniela and Francesconi, Enrico and Peters, Wim}, month = may, title = {An Evaluation of Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain}, booktitle = {Proceedings of the LREC 2008 Workshop on Semantic Processing of Legal Texts}, year = {2008}, pages = {23-32}, address = {Marrakech, Morocco}, url = {https://ke-tud.github.io/publications/papers/LREC2008.pdf}, opturl = {http://www.lrec-conf.org/proceedings/lrec2008/workshops/W9_Proceedings.pdf}, }
Eneldo Loza Mencía, and Johannes Fürnkranz

Pairwise Learning of Multilabel Classifications with Perceptrons

In Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IJCNN-08) , May 2008

#multi-label #pairwise learning #neural networks

Abs Bib DOI URL

Multiclass multilabel perceptrons (MMP) have been proposed as an efficient incremental training algorithm for addressing a multilabel prediction task with a team of perceptrons. The key idea is to train one binary classifier per label, as is typically done for addressing multilabel problems, but to make the training signal dependent on the performance of the whole ensemble. In this paper, we propose an alternative technique that is based on a pairwise approach, i.e., we incrementally train a perceptron for each pair of classes. Our evaluation on four multilabel datasets shows that the multilabel pairwise perceptron (MLPP) algorithm yields substantial improvements over MMP in terms of ranking quality and overfitting resistance, while maintaining its efficiency. Despite the quadratic increase in the number of perceptrons that have to be trained, the increase in computational complexity is bounded by the average number of labels per training example.
@inproceedings{jf:IJCNN-08, author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, title = {Pairwise Learning of Multilabel Classifications with Perceptrons}, booktitle = {Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IJCNN-08)}, year = {2008}, pages = {2900--2907}, organization = {IEEE}, address = {Hong Kong}, isbn = {978-1-4244-1821-3}, url = {https://ke-tud.github.io/publications/papers/loza08MLPP.pdf}, doi = {10.1109/IJCNN.2008.4634206}, }

Eneldo Loza Mencía, Sang-Hyeun Park, and Johannes Fürnkranz

Advances in Efficient Pairwise Multilabel Classification

May 2008

Bib URL

@techreport{jf:TUD-KE-2008-06,
  author = {Loza Menc{\'{i}}a, Eneldo and Park, Sang-Hyeun and F{\"{u}}rnkranz, Johannes},
  title = {Advances in Efficient Pairwise Multilabel Classification},
  number = {TUD-KE-2008-06},
  year = {2008},
  institution = {TU Darmstadt, Knowledge Engineering Group},
  url = {https://ke-tud.github.io/publications/reports/tud-ke-2008-06.pdf}
}

2007

Eneldo Loza Mencía, and Johannes Fürnkranz

An Evaluation of Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain

In Proceedings of the LWA 2007: Lernen - Wissen - Adaption , Sep 2007

#multi-label #scalability

Abs Bib URL

In this paper we evaluate the performance of multilabel classification algorithms on two classification tasks related to documents of the EUR-Lex database of legal documents of the European Union. It permits different settings of large-scale multilabel problems with up to 4000 classes with the same underlying documents. We compared the well known one-against-all approach (OAA) and its recently proposed improvement, the multiclass multilabel perceptron algorithm (MMP), which modifies the OAA ensemble by respecting dependencies between the base classifiers in the training protocol of the classifier ensemble. Both use the simple but very efficient perceptron algorithm as underlying classifier. This makes them very suitable for large-scale multilabel classification problems, in particular when the number of classes is high. Our results on the EUR-Lex database confirm that the MMP algorithm has a better response to an increasing number of classes than the one-against-all approach. We also show that it is principally possible to efficiently and effectively handle very large multilabel problems.
@inproceedings{jf:LWA-07-Loza, author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, editor = {Hinneburg, Alexander}, month = sep, title = {An Evaluation of Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain}, booktitle = {Proceedings of the LWA 2007: Lernen - Wissen - Adaption}, year = {2007}, pages = {126--132}, address = {Halle, Germany}, isbn = {978-3-86010-907-6}, url = {https://ke-tud.github.io/publications/papers/LWA07.pdf}, opturl = {http://lwa07.informatik.uni-halle.de/downloads/lwa07.pdf}, ee = {http://lwa07.informatik.uni-halle.de/downloads/lwa07.pdf}, }
Eneldo Loza Mencía, and Johannes Fürnkranz

Pairwise Learning of Multilabel Classifications with Perceptrons

Sep 2007

Abs Bib URL

Multiclass Multilabel Perceptrons (MMP) are an efficient incremental algorithm for training a team of perceptrons for a multilabel prediction task. The key idea is to train one binary classifier per label, as is typically done for addressing multilabel problems, but to make the training signal dependent on the performance of the whole ensemble. In this paper, we propose an alternative approach that is based on a pairwise approach, i.e., we incrementally train a perceptron for each pair of classes. An evaluation on the Reuters 2000 (RCV1) data shows that our multilabel pairwise perceptron (MLPP) algorithm yields substantial improvements over MMP in terms of ranking quality and overfitting resistance, while maintaining its efficiency. Despite the quadratic increase in the number of perceptrons that have to be trained, the increase in computational complexity is bounded by the average number of labels per training example.
@techreport{jf:TUD-KE-2007-05, author = {Loza Menc{\'{i}}a, Eneldo and F{\"{u}}rnkranz, Johannes}, title = {Pairwise Learning of Multilabel Classifications with Perceptrons}, number = {TUD-KE-2007-05}, year = {2007}, institution = {TU Darmstadt, Knowledge Engineering Group}, url = {https://ke-tud.github.io/publications/reports/tud-ke-2007-05.pdf} }

2006

Eneldo Loza Mencía

Paarweises Lernen von Multilabel-Klassifikationen mit dem Perzeptron-Algorithmus

TU Darmstadt, Knowledge Engineering Group , Mar 2006

Diplomarbeit

Bib URL

@mastersthesis{da:Loza,
  author = {Loza Menc{\'{i}}a, Eneldo},
  month = mar,
  title = {Paarweises Lernen von Multilabel-Klassifikationen mit dem Perzeptron-Algorithmus},
  year = {2006},
  school = {TU Darmstadt, Knowledge Engineering Group},
  note = {Diplomarbeit},
  url = {https://ke-tud.github.io/lehre/arbeiten/diplom/2006/Loza_Eneldo.pdf},
  betreuer = {JF}
}