# PIP-367: Propose a Contributor Repository for Pulsar

# Background knowledge

There are now many non-core, controversial features being added to Pulsar. The demand for these features does exist, but
continuously adding too many features to Pulsar can make it more bloated and blur the semantics of some functionalities
provided by Pulsar.

Here are some examples I encountered while participating in the community.

Some are questioned for potentially affecting the semantics of Pulsar, but they are ultimately added to Pulsar after
thorough discussions and proper processes;
> Pulsar is being used by many legacy systems and changing its semantics for specific usecases without considering
> consequences are creating a lot of pain and incompatibility problems for other existing systems and let's avoid doing
> it as we are struggling with such changes and
> breaking compatibility or changing semantics are just not acceptable. [0]

> As for PIP-321 getting in without a consensus, I was one who had concerns with it (and still think poorly of it), but
> I don't think it was decided in violation to the rules. [1]

Some, after discussion, were abstracted and simplified, leaving the implementation details to the users. [2]

Additionally, when contributors proposed to contribute more features to the Pulsar Client, they encountered some
skepticism. Although these concerns were addressed after discussions, they also reflected the ongoing dissatisfaction
with the bloated API of Pulsar.
> IMO the surface area the client already exposes is way too big. It feels any person that wanted to scratch their own
> itch just added their feature without holding back. This needs to be maintained.
> In OTel they have a contrib repo where people can contribute their plugin . In your case it's contributing a listener
> implementation. [3]

The suggestion of the OpenTelemetry contrib repository as a final proposal might be a good solution to this problem. It
can prevent too many new features from being added to Pulsar, which would make Pulsar's semantics vague, giving it a
kind of hodgepodge feeling. At the same time, it can also enrich the ecosystem of Pulsar.
Only vital and critical features will be included in the main repository of Pulsar, while other personalized
requirements should be placed in contributor repositories, similar to the OpenTelemetry contributor repository
approach. [4]

# Motivation

Enrich the Pulsar ecosystem by adding a contrib repository for maintaining non-core requirements.

The main differences between Pulsar's main repository and contributor repositories typically include the following
points:

**Main Repository**

- **Core Components**: The main repository houses the core functionalities and critical components of the Pulsar
  project, which form the foundation and are essential for all users.
- **Stability**: The code within the main repository is usually subjected to rigorous testing and review to ensure its
  stability and reliability.
- **Maintenance**: The main repository is maintained and updated by the project's core maintenance team.
- **Integration**: The code and features from the main repository are typically integrated into Pulsar's official
  release versions.
- **Documentation**: The main repository is complemented by comprehensive documentation and examples on the official
  website, aiding users in understanding and utilizing the project.

**Contributor Repository**

In contrast, contributor repositories might include:

- **Experimental Features**: These could be in the early stages of development and not yet mature enough to be
  incorporated into the main repository.
- **Non-Core Features**: Features that are useful for some users but not necessary for all, thus not fitting into the
  main repository.
- **Optimization for Specific Scenarios**: Tailored optimizations for particular use cases or environments, which may
  not be universally applicable.
- **Community-Driven**: Typically maintained by community members, these repositories may encompass a variety of
  perspectives and implementation approaches.
- **Diversity**: Contributor repositories can offer multiple implementations, allowing users to select based on their
  specific needs.
- **Risk**: The use of code from contributor repositories might carry higher risks due to potentially less stringent
  testing compared to the main repository.
- **Flexibility**: These repositories allow for faster iteration and experimentation, not being bound by the release
  cycle and stability requirements of the main repository.
- **Gradual Integration**: Mature features from contributor repositories are allowed to be gradually integrated into the
  main repository. This usually necessitates a series of reviews and tests to ensure they meet the quality standards of
  the main repository.
- **Community Participation**: Encourages community members to contribute and expresses gratitude to all those who have
  contributed.
- **Documentation**: Focuses on maintaining and showcasing newly introduced features to enhance transparency and
  encourage community involvement and contribution. The documentation will exclusively contain information on new
  features, including feature descriptions, development progress, and a list of contributors, to acknowledge their
  contributions to the project.

# Goals

## In Scope

### Requirement Assessment:

- Discuss the necessity of establishing a contributor repository and analyze its potential benefits to project
  development and community engagement.

### Project Affiliation:

- Evaluate and decide whether to make this repository an official subproject of Apache Pulsar for better management and
  promotion.

### Process Design:

- If the community decides to establish a contributor repository, we will collaboratively design and determine the
  contribution process to ensure it is efficient and easy for new contributors to follow.

### Maintenance Strategy:

- Develop maintenance processes and strategies, clarifying the responsibilities of maintainers, code review standards,
  and quality assurance measures.

## Out of Scope

- **Core Feature Decisions**: Deciding which features are core and which are non-core contributions is typically
  determined by project maintainers and community leaders based on the project roadmap and vision, and is not within the
  direct discussion scope of the contributor repository.

- **Feature Migration**: Migrating non-core or semantically inconsistent features of Pulsar out of the core repository
  involves in-depth analysis and decision-making of the existing codebase, usually handled by the core maintenance team.

# High Level Design

Provide a non-core code maintenance repository to collect plugin implementations, personalized features, experimental
features, and user best practices.

- This repository will not be released
- After iteration, mature features or plugins will be gradually merged into the main branch.

# Detailed Design

## Design & Implementation Details

The contributor repository is different from Pulsar's main contribution process. The contributor repository should take
a more flexible approach to handle the integration of new features:

1. **Custom interface implementation**:

- For newly added custom interface implementations, if they do not cause functional conflicts, they can be directly
  merged into the main branch after completing the code review.

2. **Potentially conflicting features**:

- For new features or optimizations that may conflict with other features, we recommend developing them in a private
  repository and maintaining a curated list in the contributor repository.

3. **Branch review and merge**:

- Maintainers and community committers review the code and decide whether to allow the plugin to be merged into the main
  branch or add experimental features to the curated list. After multiple iterations, widely used and mature features
  can be tried to be merged into the Pulsar main repository.

4. **User-defined integration**:

- Users can select the required functions according to the description in the function document and integrate them into
  their own fork for customized release.

5. **Version release**

- Contributor repositories will not be released, users need to release them by themselves.

For specific repository design, please refer to: https://github.com/StevenLuMT/pulsar-java-contrib/blob/stable/README.md?plain=1

# Alternatives

<!--
If there are alternatives that were already considered by the authors or, after the discussion, by the community, and were rejected, please list them here along with the reason why they were rejected.
-->

# General Notes

# Links

[0] - https://lists.apache.org/thread/87qfp8ht5s0fvw2y4t3j9yzgfmdzmcnz

[1] - https://lists.apache.org/thread/gzx4j9q0xdtcvrfvvq72t9tm2rt9h3r7

[2] - https://github.com/apache/pulsar/pull/22861

[3] - https://github.com/apache/pulsar/pull/22902#issuecomment-2168778056

[4] - https://github.com/open-telemetry/opentelemetry-java-contrib

* Mailing List discussion thread: https://lists.apache.org/thread/75y70j6pqw15271b0tq8j637tb02mrwo
* Mailing List voting thread: https://lists.apache.org/thread/td0j8l1c3l93nny0m5smnsdmb91j1n2y
